♻ ★ Cross-Category Functional Grasp Transfer

Rina Wu, Tianqiang Zhu, Xiangbo Lin, Yi Sun

Generating grasps for a dexterous hand often requires numerous grasping annotations. However, annotating high DoF dexterous hand poses is quite challenging. Especially for functional grasps, requiring the hand to grasp the object in a specific pose to facilitate subsequent manipulations. This prompts us to explore how people achieve manipulations on new objects based on past grasp experiences. We find that when grasping new items, people are adept at discovering and leveraging various similarities between objects, including shape, layout, and grasp type. Considering this, we analyze and collect grasp-related similarity relationships among 51 common tool-like object categories and annotate semantic grasp representation for 1768 objects. These objects are connected through similarities to form a knowledge graph, which helps infer our proposed cross-category functional grasp synthesis. Through extensive experiments, we demonstrate that the grasp-related knowledge indeed contributed to achieving functional grasp transfer across unknown or entirely new categories of objects.

♻ ★ Guiding Reinforcement Learning with Incomplete System Dynamics IROS 2024

Shuyuan Wang, Jingliang Duan, Nathan P. Lawrence, Philip D. Loewen, Michael G. Forbes, R. Bhushan Gopaluni, Lixian Zhang

Model-free reinforcement learning (RL) is inherently a reactive method, operating under the assumption that it starts with no prior knowledge of the system and entirely depends on trial-and-error for learning. This approach faces several challenges, such as poor sample efficiency, generalization, and the need for well-designed reward functions to guide learning effectively. On the other hand, controllers based on complete system dynamics do not require data. This paper addresses the intermediate situation where there is not enough model information for complete controller design, but there is enough to suggest that a model-free approach is not the best approach either. By carefully decoupling known and unknown information about the system dynamics, we obtain an embedded controller guided by our partial model and thus improve the learning efficiency of an RL-enhanced approach. A modular design allows us to deploy mainstream RL algorithms to refine the policy. Simulation results show that our method significantly improves sample efficiency compared with standard RL methods on continuous control tasks, and also offers enhanced performance over traditional control approaches. Experiments on a real ground vehicle also validate the performance of our method, including generalization and robustness.

comment: Accepted to IROS 2024

♻ ★ Effective Finite Time Stability Control for Human-Machine Shared Vehicle Following System

Zihan Wang, Mengran Li, Ronghui Zhang, Jing Zhao, Chuan Hu, Xiaolei Ma, Zhijun Qiu

With the development of intelligent connected vehicle technology, human-machine shared control has gained popularity in vehicle following due to its effectiveness in driver assistance. However, traditional vehicle following systems struggle to maintain stability when driver reaction time fluctuates, as these variations require different levels of system intervention. To address this issue, the proposed human-machine shared vehicle following assistance system (HM-VFAS) integrates driver outputs under various states with the assistance system. The system employs an intelligent driver model that accounts for reaction time delays, simulating time-varying driver outputs. A control authority allocation strategy is designed to dynamically adjust the level of intervention based on real-time driver state assessment. To handle instability from driver authority switching, the proposed solution includes a two-layer adaptive finite time sliding mode controller (A-FTSMC). The first layer is an integral sliding mode adaptive controller that ensures robustness by compensating for uncertainties in the driver output. The second layer is a fast non-singular terminal sliding mode controller designed to accelerate convergence for rapid stabilization. Using real driver videos as inputs, the performance of the HM-VFAS was evaluated. Results show that the proposed control strategy maintains a safe distance under time-varying driver states, with the actual acceleration error relative to the target acceleration maintained within 0.5m/s~2 and the maximum acceleration error reduced by 1.2m/s~2. Compared to traditional controllers, the A-FTSMC controller offers faster convergence and less vibration, reducing the stabilization time by 27.3%.

♻ ★ Exploiting Data Centres and Local Energy Communities Synergies for Market Participation

Ángel Paredes, Yihong Zhou, Chaimaa Essayeh, José A. Aguado, Thomas Morstyn

The evolving energy landscape has propelled energy communities to the forefront of modern energy management. However, existing research has yet to explore the potential synergies between data centres and energy communities, necessitating an assessment on their collective capabilities for cost efficiency, waste heat optimisation, and market participation. This paper presents a mixed integer linear programming model to assess the collaborative performance of energy communities, data centres and energy markets. The evaluation focuses on the efficient use of waste heat and the flexibility of job scheduling while minimising system energy costs and maintaining quality of service requirements for data centres. Our results, based on realistic profiles of an energy community and a data centre, showcase significant benefits of these synergies, with a 38% reduction in operating costs and an 87% decrease in heat demand.

comment: Accepted at IEEE PES ISGT Europe 2024

♻ ★ Guiding Reinforcement Learning with Incomplete System Dynamics IROS 2024

Shuyuan Wang, Jingliang Duan, Nathan P. Lawrence, Philip D. Loewen, Michael G. Forbes, R. Bhushan Gopaluni, Lixian Zhang

Model-free reinforcement learning (RL) is inherently a reactive method, operating under the assumption that it starts with no prior knowledge of the system and entirely depends on trial-and-error for learning. This approach faces several challenges, such as poor sample efficiency, generalization, and the need for well-designed reward functions to guide learning effectively. On the other hand, controllers based on complete system dynamics do not require data. This paper addresses the intermediate situation where there is not enough model information for complete controller design, but there is enough to suggest that a model-free approach is not the best approach either. By carefully decoupling known and unknown information about the system dynamics, we obtain an embedded controller guided by our partial model and thus improve the learning efficiency of an RL-enhanced approach. A modular design allows us to deploy mainstream RL algorithms to refine the policy. Simulation results show that our method significantly improves sample efficiency compared with standard RL methods on continuous control tasks, and also offers enhanced performance over traditional control approaches. Experiments on a real ground vehicle also validate the performance of our method, including generalization and robustness.

comment: Accepted to IROS 2024

♻ ★ Effective Finite Time Stability Control for Human-Machine Shared Vehicle Following System

Zihan Wang, Mengran Li, Ronghui Zhang, Jing Zhao, Chuan Hu, Xiaolei Ma, Zhijun Qiu

With the development of intelligent connected vehicle technology, human-machine shared control has gained popularity in vehicle following due to its effectiveness in driver assistance. However, traditional vehicle following systems struggle to maintain stability when driver reaction time fluctuates, as these variations require different levels of system intervention. To address this issue, the proposed human-machine shared vehicle following assistance system (HM-VFAS) integrates driver outputs under various states with the assistance system. The system employs an intelligent driver model that accounts for reaction time delays, simulating time-varying driver outputs. A control authority allocation strategy is designed to dynamically adjust the level of intervention based on real-time driver state assessment. To handle instability from driver authority switching, the proposed solution includes a two-layer adaptive finite time sliding mode controller (A-FTSMC). The first layer is an integral sliding mode adaptive controller that ensures robustness by compensating for uncertainties in the driver output. The second layer is a fast non-singular terminal sliding mode controller designed to accelerate convergence for rapid stabilization. Using real driver videos as inputs, the performance of the HM-VFAS was evaluated. Results show that the proposed control strategy maintains a safe distance under time-varying driver states, with the actual acceleration error relative to the target acceleration maintained within 0.5m/s~2 and the maximum acceleration error reduced by 1.2m/s~2. Compared to traditional controllers, the A-FTSMC controller offers faster convergence and less vibration, reducing the stabilization time by 27.3%.

♻ ★ Exploiting Data Centres and Local Energy Communities Synergies for Market Participation

Ángel Paredes, Yihong Zhou, Chaimaa Essayeh, José A. Aguado, Thomas Morstyn

The evolving energy landscape has propelled energy communities to the forefront of modern energy management. However, existing research has yet to explore the potential synergies between data centres and energy communities, necessitating an assessment on their collective capabilities for cost efficiency, waste heat optimisation, and market participation. This paper presents a mixed integer linear programming model to assess the collaborative performance of energy communities, data centres and energy markets. The evaluation focuses on the efficient use of waste heat and the flexibility of job scheduling while minimising system energy costs and maintaining quality of service requirements for data centres. Our results, based on realistic profiles of an energy community and a data centre, showcase significant benefits of these synergies, with a 38% reduction in operating costs and an 87% decrease in heat demand.

comment: Accepted at IEEE PES ISGT Europe 2024

♻ ★ Guiding Reinforcement Learning with Incomplete System Dynamics IROS 2024

Shuyuan Wang, Jingliang Duan, Nathan P. Lawrence, Philip D. Loewen, Michael G. Forbes, R. Bhushan Gopaluni, Lixian Zhang

Model-free reinforcement learning (RL) is inherently a reactive method, operating under the assumption that it starts with no prior knowledge of the system and entirely depends on trial-and-error for learning. This approach faces several challenges, such as poor sample efficiency, generalization, and the need for well-designed reward functions to guide learning effectively. On the other hand, controllers based on complete system dynamics do not require data. This paper addresses the intermediate situation where there is not enough model information for complete controller design, but there is enough to suggest that a model-free approach is not the best approach either. By carefully decoupling known and unknown information about the system dynamics, we obtain an embedded controller guided by our partial model and thus improve the learning efficiency of an RL-enhanced approach. A modular design allows us to deploy mainstream RL algorithms to refine the policy. Simulation results show that our method significantly improves sample efficiency compared with standard RL methods on continuous control tasks, and also offers enhanced performance over traditional control approaches. Experiments on a real ground vehicle also validate the performance of our method, including generalization and robustness.

comment: Accepted to IROS 2024

★ DynamicCity: Large-Scale LiDAR Generation from Dynamic Scenes

Hengwei Bian, Lingdong Kong, Haozhe Xie, Liang Pan, Yu Qiao, Ziwei Liu

LiDAR scene generation has been developing rapidly recently. However, existing methods primarily focus on generating static and single-frame scenes, overlooking the inherently dynamic nature of real-world driving environments. In this work, we introduce DynamicCity, a novel 4D LiDAR generation framework capable of generating large-scale, high-quality LiDAR scenes that capture the temporal evolution of dynamic environments. DynamicCity mainly consists of two key models. 1) A VAE model for learning HexPlane as the compact 4D representation. Instead of using naive averaging operations, DynamicCity employs a novel Projection Module to effectively compress 4D LiDAR features into six 2D feature maps for HexPlane construction, which significantly enhances HexPlane fitting quality (up to 12.56 mIoU gain). Furthermore, we utilize an Expansion & Squeeze Strategy to reconstruct 3D feature volumes in parallel, which improves both network training efficiency and reconstruction accuracy than naively querying each 3D point (up to 7.05 mIoU gain, 2.06x training speedup, and 70.84% memory reduction). 2) A DiT-based diffusion model for HexPlane generation. To make HexPlane feasible for DiT generation, a Padded Rollout Operation is proposed to reorganize all six feature planes of the HexPlane as a squared 2D feature map. In particular, various conditions could be introduced in the diffusion or sampling process, supporting versatile 4D generation applications, such as trajectory- and command-driven generation, inpainting, and layout-conditioned generation. Extensive experiments on the CarlaSC and Waymo datasets demonstrate that DynamicCity significantly outperforms existing state-of-the-art 4D LiDAR generation methods across multiple metrics. The code will be released to facilitate future research.

comment: Preprint; 29 pages, 15 figures, 7 tables; Project Page at https://dynamic-city.github.io/

★ SPIRE: Synergistic Planning, Imitation, and Reinforcement Learning for Long-Horizon Manipulation

Zihan Zhou, Animesh Garg, Dieter Fox, Caelan Garrett, Ajay Mandlekar

Robot learning has proven to be a general and effective technique for programming manipulators. Imitation learning is able to teach robots solely from human demonstrations but is bottlenecked by the capabilities of the demonstrations. Reinforcement learning uses exploration to discover better behaviors; however, the space of possible improvements can be too large to start from scratch. And for both techniques, the learning difficulty increases proportional to the length of the manipulation task. Accounting for this, we propose SPIRE, a system that first uses Task and Motion Planning (TAMP) to decompose tasks into smaller learning subproblems and second combines imitation and reinforcement learning to maximize their strengths. We develop novel strategies to train learning agents when deployed in the context of a planning system. We evaluate SPIRE on a suite of long-horizon and contact-rich robot manipulation problems. We find that SPIRE outperforms prior approaches that integrate imitation learning, reinforcement learning, and planning by 35% to 50% in average task performance, is 6 times more data efficient in the number of human demonstrations needed to train proficient agents, and learns to complete tasks nearly twice as efficiently. View https://sites.google.com/view/spire-corl-2024 for more details.

comment: Conference on Robot Learning (CoRL) 2024

★ A Pipeline for Segmenting and Structuring RGB-D Data for Robotics Applications

Zhiwu Zheng, Lauren Mentzer, Berk Iskender, Michael Price, Colm Prendergast, Audren Cloitre

We introduce a novel pipeline for segmenting and structuring color and depth (RGB-D) data. Existing processing pipelines for RGB-D data have focused on extracting geometric information alone. This approach precludes the development of more advanced robotic navigation and manipulation algorithms, which benefit from a semantic understanding of their environment. Our pipeline can segment RGB-D data into accurate semantic masks. These masks are then used to fuse raw captured point clouds into semantically separated point clouds. We store this information using the Universal Scene Description (USD) file format, a format suitable for easy querying by downstream robotics algorithms, human-friendly visualization, and robotics simulation.

★ Robust Two-View Geometry Estimation with Implicit Differentiation IROS 2024

Vladislav Pyatov, Iaroslav Koshelev, Stamatis Lefkimmiatis

We present a novel two-view geometry estimation framework which is based on a differentiable robust loss function fitting. We propose to treat the robust fundamental matrix estimation as an implicit layer, which allows us to avoid backpropagation through time and significantly improves the numerical stability. To take full advantage of the information from the feature matching stage we incorporate learnable weights that depend on the matching confidences. In this way our solution brings together feature extraction, matching and two-view geometry estimation in a unified end-to-end trainable pipeline. We evaluate our approach on the camera pose estimation task in both outdoor and indoor scenarios. The experiments on several datasets show that the proposed method outperforms both classic and learning-based state-of-the-art methods by a large margin. The project webpage is available at: https://github.com/VladPyatov/ihls

comment: IROS 2024 Accepted

★ Reconfigurable Hydrostatics: Toward Multifunctional and Powerful Wearable Robotics

Jeff Denis, Frederic Laberge, Jean-Sebastien Plante, Alexandre Girard

Wearable and locomotive robot designers face multiple challenges when choosing actuation. Traditional fully actuated designs using electric motors are multifunctional but oversized and inefficient for bearing conservative loads and for being backdrivable. Alternatively, quasi-passive and underactuated designs reduce the size of motorization and energy storage, but are often designed for specific tasks. Designers of versatile and stronger wearable robots will face these challenges unless future actuators become very torque-dense, backdrivable and efficient. This paper explores a design paradigm for addressing this issue: reconfigurable hydrostatics. We show that a hydrostatic actuator can integrate a passive force mechanism and a sharing mechanism in the fluid domain and still be multifunctional. First, an analytical study compares how these two mechanisms can relax the motorization requirements in the context of a load-bearing exoskeleton. Then, the hydrostatic concept integrating these two mechanisms using hydraulic components is presented. A case study analysis shows the mass/efficiency/inertia benefits of the concept over a fully actuated one. Then, the feasibility of the concept is partially validated with a proof-of-concept that actuates the knees of an exoskeleton. The experiments show that it can track the vertical ground reaction force (GRF) profiles of walking, running, squatting, and jumping, and that the energy consumption is 6x lower. The transient force behaviors due to switching from one leg to the other are also analyzed along with some mitigation to improve them.

★ Gaussian Process Distance Fields Obstacle and Ground Constraints for Safe Navigation

Monisha Mushtary Uttsha, Cedric Le Gentil, Lan Wu, Teresa Vidal-Calleja

Navigating cluttered environments is a challenging task for any mobile system. Existing approaches for ground-based mobile systems primarily focus on small wheeled robots, which face minimal constraints with overhanging obstacles and cannot manage steps or stairs, making the problem effectively 2D. However, navigation for legged robots (or even humans) has to consider an extra dimension. This paper proposes a tailored scene representation coupled with an advanced trajectory optimisation algorithm to enable safe navigation. Our 3D navigation approach is suitable for any ground-based mobile robot, whether wheeled or legged, as well as for human assistance. Given a 3D point cloud of the scene and the segmentation of the ground and non-ground points, we formulate two Gaussian Process distance fields to ensure a collision-free path and maintain distance to the ground constraints. Our method adeptly handles uneven terrain, steps, and overhanging objects through an innovative use of a quadtree structure, constructing a multi-resolution map of the free space and its connectivity graph based on a 2D projection of the relevant scene. Evaluations with both synthetic and real-world datasets demonstrate that this approach provides safe and smooth paths, accommodating a wide range of ground-based mobile systems.

★ Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Models

Nils Blank, Moritz Reuss, Marcel Rühle, Ömer Erdinç Yağmurlu, Fabian Wenzel, Oier Mees, Rudolf Lioutikov

A central challenge towards developing robots that can relate human language to their perception and actions is the scarcity of natural language annotations in diverse robot datasets. Moreover, robot policies that follow natural language instructions are typically trained on either templated language or expensive human-labeled instructions, hindering their scalability. To this end, we introduce NILS: Natural language Instruction Labeling for Scalability. NILS automatically labels uncurated, long-horizon robot data at scale in a zero-shot manner without any human intervention. NILS combines pretrained vision-language foundation models in order to detect objects in a scene, detect object-centric changes, segment tasks from large datasets of unlabelled interaction data and ultimately label behavior datasets. Evaluations on BridgeV2, Fractal, and a kitchen play dataset show that NILS can autonomously annotate diverse robot demonstrations of unlabeled and unstructured datasets while alleviating several shortcomings of crowdsourced human annotations, such as low data quality and diversity. We use NILS to label over 115k trajectories obtained from over 430 hours of robot data. We open-source our auto-labeling code and generated annotations on our website: http://robottasklabeling.github.io.

comment: Project Website at https://robottasklabeling.github.io/

★ Multi-Layered Safety of Redundant Robot Manipulators via Task-Oriented Planning and Control

Xinyu Jia, Wenxin Wang, Jun Yang, Yongping Pan, Haoyong Yu

Ensuring safety is crucial to promote the application of robot manipulators in open workspace. Factors such as sensor errors or unpredictable collisions make the environment full of uncertainties. In this work, we investigate these potential safety challenges on redundant robot manipulators, and propose a task-oriented planning and control framework to achieve multi-layered safety while maintaining efficient task execution. Our approach consists of two main parts: a task-oriented trajectory planner based on multiple-shooting model predictive control method, and a torque controller that allows safe and efficient collision reaction using only proprioceptive data. Through extensive simulations and real-hardware experiments, we demonstrate that the proposed framework can effectively handle uncertain static or dynamic obstacles, and perform disturbance resistance in manipulation tasks when unforeseen contacts occur. All code will be open-sourced to benefit the community.

comment: 7 pages, 8 figures. This work has been submitted to the IEEE for possible publication

★ Towards Safer Planetary Exploration: A Hybrid Architecture for Terrain Traversability Analysis in Mars Rovers

Achille Chiuchiarelli, Giacomo Franchini, Francesco Messina, Marcello Chiaberge

The field of autonomous navigation for unmanned ground vehicles (UGVs) is in continuous growth and increasing levels of autonomy have been reached in the last few years. However, the task becomes more challenging when the focus is on the exploration of planet surfaces such as Mars. In those situations, UGVs are forced to navigate through unstable and rugged terrains which, inevitably, open the vehicle to more hazards, accidents, and, in extreme cases, complete mission failure. The paper addresses the challenges of autonomous navigation for unmanned ground vehicles in planetary exploration, particularly on Mars, introducing a hybrid architecture for terrain traversability analysis that combines two approaches: appearance-based and geometry-based. The appearance-based method uses semantic segmentation via deep neural networks to classify different terrain types. This is further refined by pixel-level terrain roughness classification obtained from the same RGB image, assigning different costs based on the physical properties of the soil. The geometry-based method complements the appearance-based approach by evaluating the terrain's geometrical features, identifying hazards that may not be detectable by the appearance-based side. The outputs of both methods are combined into a comprehensive hybrid cost map. The proposed architecture was trained on synthetic datasets and developed as a ROS2 application to integrate into broader autonomous navigation systems for harsh environments. Simulations have been performed in Unity, showing the ability of the method to assess online traversability analysis.

★ Markov Potential Game with Final-time Reach-Avoid Objectives

Sarah H. Q. Li, Abraham P. Vinod

We formulate a Markov potential game with final-time reach-avoid objectives by integrating potential game theory with stochastic reach-avoid control. Our focus is on multi-player trajectory planning where players maximize the same multi-player reach-avoid objective: the probability of all participants reaching their designated target states by a specified time, while avoiding collisions with one another. Existing approaches require centralized computation of actions via a global policy, which may have prohibitively expensive communication costs. Instead, we focus on approximations of the global policy via local state feedback policies. First, we adapt the recursive single player reach-avoid value iteration to the multi-player framework with local policies, and show that the same recursion holds on the joint state space. To find each player's optimal local policy, the multi-player reach-avoid value function is projected from the joint state to the local state using the other players' occupancy measures. Then, we propose an iterative best response scheme for the multi-player value iteration to converge to a pure Nash equilibrium. We demonstrate the utility of our approach in finding collision-free policies for multi-player motion planning in simulation.

comment: 8 pages, 2 figures

★ Human-Robot Collaboration System Setup for Weed Harvesting Scenarios in Aquatic Lakes IROS 2024

Ahmed H. Elsayed, Andrej Lejman, Frederic Stahl

Artificial Water Bodies (AWBs) are human-made and require continuous monitoring due to their artificial biological processes. These systems necessitate regular maintenance to manage their ecosystems effectively. Unmanned Surface Vehicle (USV) offers a collaborative approach for monitoring these environments, working alongside human operators such as boat skippers to identify specific locations. This paper discusses a weed harvesting scenario, demonstrating how human-robot collaboration can be achieved, supported by preliminary results. The USV mainly utilises multibeam SOund NAvigation and Ranging (SONAR) for underwater weed monitoring, showing promising outcomes in these scenarios.

comment: 3 pages, 5 figures. This paper was accepted for poster presentation at IROS 2024 Workshop on Maritime Heteregenous Unmanned Robotic Systems (MHURS)

★ Incremental Learning of Affordances using Markov Logic Networks

George Potter, Gertjan Burghouts, Joris Sijs

Affordances enable robots to have a semantic understanding of their surroundings. This allows them to have more acting flexibility when completing a given task. Capturing object affordances in a machine learning model is a difficult task, because of their dependence on contextual information. Markov Logic Networks (MLN) combine probabilistic reasoning with logic that is able to capture such context. Mobile robots operate in partially known environments wherein unseen object affordances can be observed. This new information must be incorporated into the existing knowledge, without having to retrain the MLN from scratch. We introduce the MLN Cumulative Learning Algorithm (MLN-CLA). MLN-CLA learns new relations in various knowledge domains by retaining knowledge and only updating the changed knowledge, for which the MLN is retrained. We show that MLN-CLA is effective for accumulative learning and zero-shot affordance inference, outperforming strong baselines.

comment: accepted at IEEE IRC 2024

★ ImDy: Human Inverse Dynamics from Imitated Observations

Xinpeng Liu, Junxuan Liang, Zili Lin, Haowen Hou, Yong-Lu Li, Cewu Lu

Inverse dynamics (ID), which aims at reproducing the driven torques from human kinematic observations, has been a critical tool for gait analysis. However, it is hindered from wider application to general motion due to its limited scalability. Conventional optimization-based ID requires expensive laboratory setups, restricting its availability. To alleviate this problem, we propose to exploit the recently progressive human motion imitation algorithms to learn human inverse dynamics in a data-driven manner. The key insight is that the human ID knowledge is implicitly possessed by motion imitators, though not directly applicable. In light of this, we devise an efficient data collection pipeline with state-of-the-art motion imitation algorithms and physics simulators, resulting in a large-scale human inverse dynamics benchmark as Imitated Dynamics (ImDy). ImDy contains over 150 hours of motion with joint torque and full-body ground reaction force data. With ImDy, we train a data-driven human inverse dynamics solver ImDyS(olver) in a fully supervised manner, which conducts ID and ground reaction force estimation simultaneously. Experiments on ImDy and real-world data demonstrate the impressive competency of ImDyS in human inverse dynamics and ground reaction force estimation. Moreover, the potential of ImDy(-S) as a fundamental motion analysis tool is exhibited with downstream applications. The project page is https://foruck.github.io/ImDy/.

comment: Yong-Lu Li and Cewu Lu are the corresponding authors

★ Integrating Large Language Models for UAV Control in Simulated Environments: A Modular Interaction Approach

Abhishek Phadke, Alihan Hadimlioglu, Tianxing Chu, Chandra N Sekharan

The intersection of LLMs (Large Language Models) and UAV (Unoccupied Aerial Vehicles) technology represents a promising field of research with the potential to enhance UAV capabilities significantly. This study explores the application of LLMs in UAV control, focusing on the opportunities for integrating advanced natural language processing into autonomous aerial systems. By enabling UAVs to interpret and respond to natural language commands, LLMs simplify the UAV control and usage, making them accessible to a broader user base and facilitating more intuitive human-machine interactions. The paper discusses several key areas where LLMs can impact UAV technology, including autonomous decision-making, dynamic mission planning, enhanced situational awareness, and improved safety protocols. Through a comprehensive review of current developments and potential future directions, this study aims to highlight how LLMs can transform UAV operations, making them more adaptable, responsive, and efficient in complex environments. A template development framework for integrating LLMs in UAV control is also described. Proof of Concept results that integrate existing LLM models and popular robotic simulation platforms are demonstrated. The findings suggest that while there are substantial technical and ethical challenges to address, integrating LLMs into UAV control holds promising implications for advancing autonomous aerial systems.

★ Energy-Optimal Planning of Waypoint-Based UAV Missions -- Does Minimum Distance Mean Minimum Energy? IROS

Nicolas Michel, Ayush Patnaik, Zhaodan Kong, Xinfan Lin

Multirotor unmanned aerial vehicle is a prevailing type of aerial robots with wide real-world applications. The energy efficiency of the robot is a critical aspect of its performance, determining the range and duration of the missions that can be performed. This paper studies the energy-optimal planning of the multirotor, which aims at finding the optimal ordering of waypoints with the minimum energy consumption for missions in 3D space. The study is performed based on a previously developed model capturing first-principle energy dynamics of the multirotor. We found that in majority of the cases (up to 95%) the solutions of the energy-optimal planning are different from those of the traditional traveling salesman problem which minimizes the total distance. The difference can be as high as 14.9%, with the average at 1.6%-3.3% and 90th percentile at 3.7%-6.5% depending on the range and number of waypoints in the mission. We then identified and explained the key features of the minimum-energy order by correlating to the underlying flight energy dynamics. It is shown that instead of minimizing the distance, coordination of vertical and horizontal motion to promote aerodynamic efficiency is the key to optimizing energy consumption.

comment: This paper has been accepted for presentation at the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2024

★ Real-time Vehicle-to-Vehicle Communication Based Network Cooperative Control System through Distributed Database and Multimodal Perception: Demonstrated in Crossroads

Xinwen Zhu, Zihao Li, Yuxuan Jiang, Jiazhen Xu, Jie Wang, Xuyang Bai

The autonomous driving industry is rapidly advancing, with Vehicle-to-Vehicle (V2V) communication systems highlighting as a key component of enhanced road safety and traffic efficiency. This paper introduces a novel Real-time Vehicle-to-Vehicle Communication Based Network Cooperative Control System (VVCCS), designed to revolutionize macro-scope traffic planning and collision avoidance in autonomous driving. Implemented on Quanser Car (Qcar) hardware platform, our system integrates the distributed databases into individual autonomous vehicles and an optional central server. We also developed a comprehensive multi-modal perception system with multi-objective tracking and radar sensing. Through a demonstration within a physical crossroad environment, our system showcases its potential to be applied in congested and complex urban environments.

comment: ICICT 2024, 18 pages

★ Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensors

Bang You, Huaping Liu

Reinforcement learning has achieved promising results on robotic control tasks but struggles to leverage information effectively from multiple sensory modalities that differ in many characteristics. Recent works construct auxiliary losses based on reconstruction or mutual information to extract joint representations from multiple sensory inputs to improve the sample efficiency and performance of reinforcement learning algorithms. However, the representations learned by these methods could capture information irrelevant to learning a policy and may degrade the performance. We argue that compressing information in the learned joint representations about raw multimodal observations is helpful, and propose a multimodal information bottleneck model to learn task-relevant joint representations from egocentric images and proprioception. Our model compresses and retains the predictive information in multimodal observations for learning a compressed joint representation, which fuses complementary information from visual and proprioceptive feedback and meanwhile filters out task-irrelevant information in raw multimodal observations. We propose to minimize the upper bound of our multimodal information bottleneck objective for computationally tractable optimization. Experimental evaluations on several challenging locomotion tasks with egocentric images and proprioception show that our method achieves better sample efficiency and zero-shot robustness to unseen white noise than leading baselines. We also empirically demonstrate that leveraging information from egocentric images and proprioception is more helpful for learning policies on locomotion tasks than solely using one single modality.

comment: 31 pages

★ Generalizable Motion Planning via Operator Learning

Sharath Matada, Luke Bhan, Yuanyuan Shi, Nikolay Atanasov

In this work, we introduce a planning neural operator (PNO) for predicting the value function of a motion planning problem. We recast value function approximation as learning a single operator from the cost function space to the value function space, which is defined by an Eikonal partial differential equation (PDE). Specifically, we recast computing value functions as learning a single operator across continuous function spaces which prove is equivalent to solving an Eikonal PDE. Through this reformulation, our learned PNO is able to generalize to new motion planning problems without retraining. Therefore, our PNO model, despite being trained with a finite number of samples at coarse resolution, inherits the zero-shot super-resolution property of neural operators. We demonstrate accurate value function approximation at 16 times the training resolution on the MovingAI lab's 2D city dataset and compare with state-of-the-art neural value function predictors on 3D scenes from the iGibson building dataset. Lastly, we investigate employing the value function output of PNO as a heuristic function to accelerate motion planning. We show theoretically that the PNO heuristic is $\epsilon$-consistent by introducing an inductive bias layer that guarantees our value functions satisfy the triangle inequality. With our heuristic, we achieve a 30% decrease in nodes visited while obtaining near optimal path lengths on the MovingAI lab 2D city dataset, compared to classical planning methods (A*, RRT*).

★ Mechanisms and Computational Design of Multi-Modal End-Effector with Force Sensing using Gated Networks

Yusuke Tanaka, Alvin Zhu, Richard Lin, Ankur Mehta, Dennis Hong

In limbed robotics, end-effectors must serve dual functions, such as both feet for locomotion and grippers for grasping, which presents design challenges. This paper introduces a multi-modal end-effector capable of transitioning between flat and line foot configurations while providing grasping capabilities. MAGPIE integrates 8-axis force sensing using proposed mechanisms with hall effect sensors, enabling both contact and tactile force measurements. We present a computational design framework for our sensing mechanism that accounts for noise and interference, allowing for desired sensitivity and force ranges and generating ideal inverse models. The hardware implementation of MAGPIE is validated through experiments, demonstrating its capability as a foot and verifying the performance of the sensing mechanisms, ideal models, and gated network-based models.

★ X-MOBILITY: End-To-End Generalizable Navigation via World Modeling

Wei Liu, Huihua Zhao, Chenran Li, Joydeep Biswas, Billy Okal, Pulkit Goyal, Yan Chang, Soha Pouya

General-purpose navigation in challenging environments remains a significant problem in robotics, with current state-of-the-art approaches facing myriad limitations. Classical approaches struggle with cluttered settings and require extensive tuning, while learning-based methods face difficulties generalizing to out-of-distribution environments. This paper introduces X-Mobility, an end-to-end generalizable navigation model that overcomes existing challenges by leveraging three key ideas. First, X-Mobility employs an auto-regressive world modeling architecture with a latent state space to capture world dynamics. Second, a diverse set of multi-head decoders enables the model to learn a rich state representation that correlates strongly with effective navigation skills. Third, by decoupling world modeling from action policy, our architecture can train effectively on a variety of data sources, both with and without expert policies: off-policy data allows the model to learn world dynamics, while on-policy data with supervisory control enables optimal action policy learning. Through extensive experiments, we demonstrate that X-Mobility not only generalizes effectively but also surpasses current state-of-the-art navigation approaches. Additionally, X-Mobility also achieves zero-shot Sim2Real transferability and shows strong potential for cross-embodiment generalization.

★ GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policy

Yixuan Wang, Guang Yin, Binghao Huang, Tarik Kelestemur, Jiuguang Wang, Yunzhu Li

Diffusion-based policies have shown remarkable capability in executing complex robotic manipulation tasks but lack explicit characterization of geometry and semantics, which often limits their ability to generalize to unseen objects and layouts. To enhance the generalization capabilities of Diffusion Policy, we introduce a novel framework that incorporates explicit spatial and semantic information via 3D semantic fields. We generate 3D descriptor fields from multi-view RGBD observations with large foundational vision models, then compare these descriptor fields against reference descriptors to obtain semantic fields. The proposed method explicitly considers geometry and semantics, enabling strong generalization capabilities in tasks requiring category-level generalization, resolving geometric ambiguities, and attention to subtle geometric details. We evaluate our method across eight tasks involving articulated objects and instances with varying shapes and textures from multiple object categories. Our method demonstrates its effectiveness by increasing Diffusion Policy's average success rate on unseen instances from 20% to 93%. Additionally, we provide a detailed analysis and visualization to interpret the sources of performance gain and explain how our method can generalize to novel instances.

comment: Accepted to Conference on Robot Learning (CoRL 2024). Project Page: https://robopil.github.io/GenDP/

♻ ★ Diffusion-Reward Adversarial Imitation Learning

Chun-Mao Lai, Hsiang-Chun Wang, Ping-Chun Hsieh, Yu-Chiang Frank Wang, Min-Hung Chen, Shao-Hua Sun

Imitation learning aims to learn a policy from observing expert demonstrations without access to reward signals from environments. Generative adversarial imitation learning (GAIL) formulates imitation learning as adversarial learning, employing a generator policy learning to imitate expert behaviors and discriminator learning to distinguish the expert demonstrations from agent trajectories. Despite its encouraging results, GAIL training is often brittle and unstable. Inspired by the recent dominance of diffusion models in generative modeling, we propose Diffusion-Reward Adversarial Imitation Learning (DRAIL), which integrates a diffusion model into GAIL, aiming to yield more robust and smoother rewards for policy learning. Specifically, we propose a diffusion discriminative classifier to construct an enhanced discriminator, and design diffusion rewards based on the classifier's output for policy learning. Extensive experiments are conducted in navigation, manipulation, and locomotion, verifying DRAIL's effectiveness compared to prior imitation learning methods. Moreover, additional experimental results demonstrate the generalizability and data efficiency of DRAIL. Visualized learned reward functions of GAIL and DRAIL suggest that DRAIL can produce more robust and smoother rewards. Project page: https://nturobotlearninglab.github.io/DRAIL/

♻ ★ Flying through Moving Gates without Full State Estimation

Ralf Römer, Tim Emmert, Angela P. Schoellig

Autonomous drone racing requires powerful perception, planning, and control and has become a benchmark and test field for autonomous, agile flight. Existing work usually assumes static race tracks with known maps, which enables offline planning of time-optimal trajectories, performing localization to the gates to reduce the drift in visual-inertial odometry (VIO) for state estimation or training learning-based methods for the particular race track and operating environment. In contrast, many real-world tasks like disaster response or delivery need to be performed in unknown and dynamic environments. To close this gap and make drone racing more robust against unseen environments and moving gates, we propose a control algorithm that does not require a race track map or VIO and uses only monocular measurements of the line of sight (LOS) to the gates. For this purpose, we adopt the law of proportional navigation (PN) to accurately fly through the gates despite gate motions or wind. We formulate the PN-informed vision-based control problem for drone racing as a constrained optimization problem and derive a closed-form optimal solution. We demonstrate through extensive simulations and real-world experiments that our method can navigate through moving gates at high speeds while being robust to different gate movements, model errors, wind, and delays.

comment: 7 pages, 6 figures

♻ ★ The Art of Imitation: Learning Long-Horizon Manipulation Tasks from Few Demonstrations

Jan Ole von Hartz, Tim Welschehold, Abhinav Valada, Joschka Boedecker

Task Parametrized Gaussian Mixture Models (TP-GMM) are a sample-efficient method for learning object-centric robot manipulation tasks. However, there are several open challenges to applying TP-GMMs in the wild. In this work, we tackle three crucial challenges synergistically. First, end-effector velocities are non-Euclidean and thus hard to model using standard GMMs. We thus propose to factorize the robot's end-effector velocity into its direction and magnitude, and model them using Riemannian GMMs. Second, we leverage the factorized velocities to segment and sequence skills from complex demonstration trajectories. Through the segmentation, we further align skill trajectories and hence leverage time as a powerful inductive bias. Third, we present a method to automatically detect relevant task parameters per skill from visual observations. Our approach enables learning complex manipulation tasks from just five demonstrations while using only RGB-D observations. Extensive experimental evaluations on RLBench demonstrate that our approach achieves state-of-the-art performance with 20-fold improved sample efficiency. Our policies generalize across different environments, object instances, and object positions, while the learned skills are reusable.

♻ ★ JointMotion: Joint Self-Supervision for Joint Motion Prediction

Royden Wagner, Omer Sahin Tas, Marvin Klemp, Carlos Fernandez

We present JointMotion, a self-supervised pre-training method for joint motion prediction in self-driving vehicles. Our method jointly optimizes a scene-level objective connecting motion and environments, and an instance-level objective to refine learned representations. Scene-level representations are learned via non-contrastive similarity learning of past motion sequences and environment context. At the instance level, we use masked autoencoding to refine multimodal polyline representations. We complement this with an adaptive pre-training decoder that enables JointMotion to generalize across different environment representations, fusion mechanisms, and dataset characteristics. Notably, our method reduces the joint final displacement error of Wayformer, HPTR, and Scene Transformer models by 3\%, 8\%, and 12\%, respectively; and enables transfer learning between the Waymo Open Motion and the Argoverse 2 Motion Forecasting datasets. Code: https://github.com/kit-mrt/future-motion

comment: CoRL'24 camera-ready

♻ ★ UniSaT: Unified-Objective Belief Model and Planner to Search for and Track Multiple Objects SC

Leonardo Santos, Brady Moon, Sebastian Scherer, Hoa Van Nguyen

Path planning for autonomous search and tracking of multiple objects is a critical problem in applications such as reconnaissance, surveillance, and data gathering. Due to the inherent competing objectives of searching for new objects while maintaining tracks for found objects, most current approaches rely on multi-objective planning methods, leaving it up to the user to tune parameters to balance between the two objectives, usually based on heuristics or trial and error. In this paper, we introduce UniSaT (Unified Search and Track), a novel unified-objective formulation for the search and track problem based on Random Finite Sets (RFS). Our approach models unknown and known objects using a combined generalized labeled multi-Bernoulli (GLMB) filter. For unseen objects, UniSaT leverages both cardinality and spatial prior distributions, allowing it to operate without prior knowledge of the exact number of objects in the search space. The planner maximizes the mutual information of this unified belief model, creating balanced search and tracking behaviors. We demonstrate our work in a simulated environment, presenting both qualitative results and quantitative improvements over a multi-objective method.

comment: 13 pages, AIAA SCITECH 2025 Forum

♻ ★ DexGrasp-Diffusion: Diffusion-based Unified Functional Grasp Synthesis Method for Multi-Dexterous Robotic Hands

Zhengshen Zhang, Lei Zhou, Chenchen Liu, Zhiyang Liu, Chengran Yuan, Sheng Guo, Ruiteng Zhao, Marcelo H. Ang Jr., Francis EH Tay

The versatility and adaptability of human grasping catalyze advancing dexterous robotic manipulation. While significant strides have been made in dexterous grasp generation, current research endeavors pivot towards optimizing object manipulation while ensuring functional integrity, emphasizing the synthesis of functional grasps following desired affordance instructions. This paper addresses the challenge of synthesizing functional grasps tailored to diverse dexterous robotic hands by proposing DexGrasp-Diffusion, an end-to-end modularized diffusion-based method. DexGrasp-Diffusion integrates MultiHandDiffuser, a novel unified data-driven diffusion model for multi-dexterous hands grasp estimation, with DexDiscriminator, which employs a Physics Discriminator and a Functional Discriminator with open-vocabulary setting to filter physically plausible functional grasps based on object affordances. The experimental evaluation conducted on the MultiDex dataset provides substantiating evidence supporting the superior performance of MultiHandDiffuser over the baseline model in terms of success rate, grasp diversity, and collision depth. Moreover, we demonstrate the capacity of DexGrasp-Diffusion to reliably generate functional grasps for household objects aligned with specific affordance instructions.

comment: 15 pages, 5 figures

♻ ★ ODTFormer: Efficient Obstacle Detection and Tracking with Stereo Cameras Based on Transformer IROS 2024

Tianye Ding, Hongyu Li, Huaizu Jiang

Obstacle detection and tracking represent a critical component in robot autonomous navigation. In this paper, we propose ODTFormer, a Transformer-based model to address both obstacle detection and tracking problems. For the detection task, our approach leverages deformable attention to construct a 3D cost volume, which is decoded progressively in the form of voxel occupancy grids. We further track the obstacles by matching the voxels between consecutive frames. The entire model can be optimized in an end-to-end manner. Through extensive experiments on DrivingStereo and KITTI benchmarks, our model achieves state-of-the-art performance in the obstacle detection task. We also report comparable accuracy to state-of-the-art obstacle tracking models while requiring only a fraction of their computation cost, typically ten-fold to twenty-fold less. The code and model weights will be publicly released.

comment: 8 pages. Accepted by IROS 2024

♻ ★ Cross-Category Functional Grasp Transfer

Rina Wu, Tianqiang Zhu, Xiangbo Lin, Yi Sun

Generating grasps for a dexterous hand often requires numerous grasping annotations. However, annotating high DoF dexterous hand poses is quite challenging. Especially for functional grasps, requiring the hand to grasp the object in a specific pose to facilitate subsequent manipulations. This prompts us to explore how people achieve manipulations on new objects based on past grasp experiences. We find that when grasping new items, people are adept at discovering and leveraging various similarities between objects, including shape, layout, and grasp type. Considering this, we analyze and collect grasp-related similarity relationships among 51 common tool-like object categories and annotate semantic grasp representation for 1768 objects. These objects are connected through similarities to form a knowledge graph, which helps infer our proposed cross-category functional grasp synthesis. Through extensive experiments, we demonstrate that the grasp-related knowledge indeed contributed to achieving functional grasp transfer across unknown or entirely new categories of objects.

♻ ★ Gaussian-Informed Continuum for Physical Property Identification and Simulation NeurIPS 2024

Junhao Cai, Yuji Yang, Weihao Yuan, Yisheng He, Zilong Dong, Liefeng Bo, Hui Cheng, Qifeng Chen

This paper studies the problem of estimating physical properties (system identification) through visual observations. To facilitate geometry-aware guidance in physical property estimation, we introduce a novel hybrid framework that leverages 3D Gaussian representation to not only capture explicit shapes but also enable the simulated continuum to render object masks as 2D shape surrogates during training. We propose a new dynamic 3D Gaussian framework based on motion factorization to recover the object as 3D Gaussian point sets across different time states. Furthermore, we develop a coarse-to-fine filling strategy to generate the density fields of the object from the Gaussian reconstruction, allowing for the extraction of object continuums along with their surfaces and the integration of Gaussian attributes into these continuums. In addition to the extracted object surfaces, the Gaussian-informed continuum also enables the rendering of object masks during simulations, serving as 2D-shape guidance for physical property estimation. Extensive experimental evaluations demonstrate that our pipeline achieves state-of-the-art performance across multiple benchmarks and metrics. Additionally, we illustrate the effectiveness of the proposed method through real-world demonstrations, showcasing its practical utility. Our project page is at https://jukgei.github.io/project/gic.

comment: 21 pages, 8 figures, NeurIPS 2024

♻ ★ Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learning

Zhecheng Yuan, Tianming Wei, Shuiqi Cheng, Gu Zhang, Yuanpei Chen, Huazhe Xu

Can we endow visuomotor robots with generalization capabilities to operate in diverse open-world scenarios? In this paper, we propose \textbf{Maniwhere}, a generalizable framework tailored for visual reinforcement learning, enabling the trained robot policies to generalize across a combination of multiple visual disturbance types. Specifically, we introduce a multi-view representation learning approach fused with Spatial Transformer Network (STN) module to capture shared semantic information and correspondences among different viewpoints. In addition, we employ a curriculum-based randomization and augmentation approach to stabilize the RL training process and strengthen the visual generalization ability. To exhibit the effectiveness of Maniwhere, we meticulously design 8 tasks encompassing articulate objects, bi-manual, and dexterous hand manipulation tasks, demonstrating Maniwhere's strong visual generalization and sim2real transfer abilities across 3 hardware platforms. Our experiments show that Maniwhere significantly outperforms existing state-of-the-art methods. Videos are provided at https://gemcollector.github.io/maniwhere/.

comment: Webpage: https://gemcollector.github.io/maniwhere/

♻ ★ Exploring Self-Supervised Skeleton-Based Human Action Recognition under Occlusions

Yifei Chen, Kunyu Peng, Alina Roitberg, David Schneider, Jiaming Zhang, Junwei Zheng, Ruiping Liu, Yufan Chen, Kailun Yang, Rainer Stiefelhagen

To integrate self-supervised skeleton-based action recognition methods into autonomous robotic systems, it is crucial to consider adverse situations involving target occlusions. Such a scenario, despite its practical relevance, is rarely addressed in existing self-supervised skeleton-based action recognition methods. To empower models with the capacity to address occlusion, we propose a simple and effective method. We first pre-train using occluded skeleton sequences, then use k-means clustering (KMeans) on sequence embeddings to group semantically similar samples. Next, we propose KNN-Imputation to fill in missing skeleton data based on the closest sample neighbors. Imputing incomplete skeleton sequences to create relatively complete sequences as input provides significant benefits to existing skeleton-based self-supervised methods. Meanwhile, building on the state-of-the-art Partial Spatio-Temporal Learning (PSTL), we introduce an Occluded Partial Spatio-Temporal Learning (OPSTL) framework. This enhancement utilizes Adaptive Spatial Masking (ASM) for better use of high-quality, intact skeletons. The new proposed method is verified on the challenging occluded versions of the NTURGB+D 60 and NTURGB+D 120. The source code is publicly available at https://github.com/cyfml/OPSTL.

comment: The source code is publicly available at https://github.com/cyfml/OPSTL

♻ ★ Interactive Distance Field Mapping and Planning to Enable Human-Robot Collaboration

Usama Ali, Lan Wu, Adrian Mueller, Fouad Sukkar, Tobias Kaupp, Teresa Vidal-Calleja

Human-robot collaborative applications require scene representations that are kept up-to-date and facilitate safe motions in dynamic scenes. In this letter, we present an interactive distance field mapping and planning (IDMP) framework that handles dynamic objects and collision avoidance through an efficient representation. We define interactive mapping and planning as the process of creating and updating the representation of the scene online while simultaneously planning and adapting the robot's actions based on that representation. The key aspect of this work is an efficient Gaussian Process field that performs incremental updates and handles dynamic objects reliably by identifying moving points via a simple and elegant formulation based on queries from a temporary latent model. In terms of mapping, IDMP is able to fuse point cloud data from single and multiple sensors, query the free space at any spatial resolution, and deal with moving objects without semantics. In terms of planning, IDMP allows seamless integration with gradient-based reactive planners facilitating dynamic obstacle avoidance for safe human-robot interactions. Our mapping performance is evaluated on both real and synthetic datasets. A comparison with similar state-of-the-art frameworks shows superior performance when handling dynamic objects and comparable or better performance in the accuracy of the computed distance and gradient field. Finally, we show how the framework can be used for fast motion planning in the presence of moving objects both in simulated and real-world scenes. An accompanying video, code, and datasets are made publicly available https://uts-ri.github.io/IDMP.

♻ ★ Real-World Robot Applications of Foundation Models: A Review

Kento Kawaharazuka, Tatsuya Matsushima, Andrew Gambardella, Jiaxian Guo, Chris Paxton, Andy Zeng

Recent developments in foundation models, like Large Language Models (LLMs) and Vision-Language Models (VLMs), trained on extensive data, facilitate flexible application across different tasks and modalities. Their impact spans various fields, including healthcare, education, and robotics. This paper provides an overview of the practical application of foundation models in real-world robotics, with a primary emphasis on the replacement of specific components within existing robot systems. The summary encompasses the perspective of input-output relationships in foundation models, as well as their role in perception, motion planning, and control within the field of robotics. This paper concludes with a discussion of future challenges and implications for practical robot applications.

♻ ★ Log-GPIS-MOP: A Unified Representation for Mapping, Odometry and Planning

Lan Wu, Ki Myung Brian Lee, Cedric Le Gentil, Teresa Vidal-Calleja

Whereas dedicated scene representations are required for each different task in conventional robotic systems, this paper demonstrates that a unified representation can be used directly for multiple key tasks. We propose the Log-Gaussian Process Implicit Surface for Mapping, Odometry and Planning (Log-GPIS-MOP): a probabilistic framework for surface reconstruction, localisation and navigation based on a unified representation. Our framework applies a logarithmic transformation to a Gaussian Process Implicit Surface (GPIS) formulation to recover a global representation that accurately captures the Euclidean distance field with gradients and, at the same time, the implicit surface. By directly estimating the distance field and its gradient through Log-GPIS inference, the proposed incremental odometry technique computes the optimal alignment of an incoming frame and fuses it globally to produce a map. Concurrently, an optimisation-based planner computes a safe collision-free path using the same Log-GPIS surface representation. We validate the proposed framework on simulated and real datasets in 2D and 3D and benchmark against the state-of-the-art approaches. Our experiments show that Log-GPIS-MOP produces competitive results in sequential odometry, surface mapping and obstacle avoidance.

♻ ★ The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewards

Sukai Huang, Nir Lipovetzky, Trevor Cohn

While Vision-Language Models (VLMs) are increasingly used to generate reward signals for training embodied agents to follow instructions, our research reveals that agents guided by VLM rewards often underperform compared to those employing only intrinsic (exploration-driven) rewards, contradicting expectations set by recent work. We hypothesize that false positive rewards -- instances where unintended trajectories are incorrectly rewarded -- are more detrimental than false negatives. Our analysis confirms this hypothesis, revealing that the widely used cosine similarity metric is prone to false positive reward estimates. To address this, we introduce BiMI ({Bi}nary {M}utual {I}nformation), a novel reward function designed to mitigate noise. BiMI significantly enhances learning efficiency across diverse and challenging embodied navigation environments. Our findings offer a nuanced understanding of how different types of reward noise impact agent learning and highlight the importance of addressing multimodal reward signal noise when training embodied agents

comment: 10 main body pages, 11 appendix pages

♻ ★ OA-MPC: Occlusion-Aware MPC for Guaranteed Safe Robot Navigation with Unseen Dynamic Obstacles

Roya Firoozi, Alexandre Mir, Gadi Sznaier Camps, Mac Schwager

For safe navigation in dynamic uncertain environments, robotic systems rely on the perception and prediction of other agents. Particularly, in occluded areas where cameras and LiDAR give no data, the robot must be able to reason about potential movements of invisible dynamic agents. This work presents a provably safe motion planning scheme for real-time navigation in an a priori unmapped environment, where occluded dynamic agents are present. Safety guarantees are provided based on reachability analysis. Forward reachable sets associated with potential occluded agents, such as pedestrians, are computed and incorporated into planning. An iterative optimization-based planner is presented that alternates between two optimizations: nonlinear Model Predictive Control (NMPC) and collision avoidance. Recursive feasibility of the MPC is guaranteed by introducing a terminal stopping constraint. The effectiveness of the proposed algorithm is demonstrated through simulation studies and hardware experiments with a TurtleBot robot. A video of experimental results is available at \url{https://youtu.be/OUnkB5Feyuk}.

♻ ★ OrionNav: Online Planning for Robot Autonomy with Context-Aware LLM and Open-Vocabulary Semantic Scene Graphs

Venkata Naren Devarakonda, Raktim Gautam Goswami, Ali Umut Kaypak, Naman Patel, Rooholla Khorrambakht, Prashanth Krishnamurthy, Farshad Khorrami

Enabling robots to autonomously navigate unknown, complex, dynamic environments and perform diverse tasks remains a fundamental challenge in developing robust autonomous physical agents. These agents must effectively perceive their surroundings while leveraging world knowledge for decision-making. Although recent approaches utilize vision-language and large language models for scene understanding and planning, they often rely on offline processing, offboard compute, make simplifying assumptions about the environment and perception, limiting real-world applicability. We present a novel framework for real-time onboard autonomous navigation in unknown environments that change over time by integrating multi-level abstraction in both perception and planning pipelines. Our system fuses data from multiple onboard sensors for localization and mapping and integrates it with open-vocabulary semantics to generate hierarchical scene graphs from continuously updated semantic object map. The LLM-based planner uses these graphs to create multi-step plans that guide low-level controllers in executing navigation tasks specified in natural language. The system's real-time operation enables the LLM to adjust its plans based on updates to the scene graph and task execution status, ensuring continuous adaptation to new situations or when the current plan cannot accomplish the task, a key advantage over static or rule-based systems. We demonstrate our system's efficacy on a quadruped navigating dynamic environments, showcasing its adaptability and robustness in diverse scenarios.

♻ ★ Piecewise Stochastic Barrier Functions

Rayan Mazouz, Frederik Baymler Mathiesen, Luca Laurenti, Morteza Lahijanian

This paper presents a novel stochastic barrier function (SBF) framework for safety analysis of stochastic systems based on piecewise (PW) functions. We first outline a general formulation of PW-SBFs. Then, we focus on PW-Constant (PWC) SBFs and show how their simplicity yields computational advantages for general stochastic systems. Specifically, we prove that synthesis of PWC-SBFs reduces to a minimax optimization problem. Then, we introduce three efficient algorithms to solve this problem, each offering distinct advantages and disadvantages. The first algorithm is based on dual linear programming (LP), which provides an exact solution to the minimax optimization problem. The second is a more scalable algorithm based on iterative counter-example guided synthesis, which involves solving two smaller LPs. The third algorithm solves the minimax problem using gradient descent, which admits even better scalability. We provide an extensive evaluation of these methods on various case studies, including neural network dynamic models, nonlinear switched systems, and high-dimensional linear systems. Our benchmarks demonstrate that PWC-SBFs outperform state-of-the-art methods, namely sum-of-squares and neural barrier functions, and can scale to eight dimensional systems.

★ GraphTeam: Facilitating Large Language Model-based Graph Analysis via Multi-Agent Collaboration

Xin Li, Qizhi Chu, Yubin Chen, Yang Liu, Yaoqi Liu, Zekai Yu, Weize Chen, Chen Qian, Chuan Shi, Cheng Yang

Graphs are widely used for modeling relational data in real-world scenarios, such as social networks and urban computing. Existing LLM-based graph analysis approaches either integrate graph neural networks (GNNs) for specific machine learning tasks, limiting their transferability, or rely solely on LLMs' internal reasoning ability, resulting in suboptimal performance. To address these limitations, we take advantage of recent advances in LLM-based agents, which have shown capabilities of utilizing external knowledge or tools for problem solving. By simulating human problem-solving strategies such as analogy and collaboration, we propose a multi-agent system based on LLMs named GraphTeam, for graph analysis. GraphTeam consists of five LLM-based agents from three modules, and the agents with different specialities can collaborate with each other to address complex problems. Specifically, (1) input-output normalization module: the question agent extracts and refines four key arguments from the original question, facilitating the problem understanding, and the answer agent organizes the results to meet the output requirement; (2) external knowledge retrieval module: we first build a knowledge base consisting of relevant documentation and experience information, and then the search agent retrieves the most relevant entries for each question. (3) problem-solving module: given the retrieved information from search agent, the coding agent uses established algorithms via programming to generate solutions, and in case the coding agent does not work, the reasoning agent will directly compute the results without programming. Extensive experiments on six graph analysis benchmarks demonstrate that GraphTeam achieves state-of-the-art performance with an average 25.85% improvement over the best baseline in terms of accuracy. The code and data are available at https://github.com/BUPT-GAMMA/GraphTeam.

★ Scalable Offline Reinforcement Learning for Mean Field Games AAMAS

Axel Brunnbauer, Julian Lemmel, Zahra Babaiee, Sophie Neubauer, Radu Grosu

Reinforcement learning algorithms for mean-field games offer a scalable framework for optimizing policies in large populations of interacting agents. Existing methods often depend on online interactions or access to system dynamics, limiting their practicality in real-world scenarios where such interactions are infeasible or difficult to model. In this paper, we present Offline Munchausen Mirror Descent (Off-MMD), a novel mean-field RL algorithm that approximates equilibrium policies in mean-field games using purely offline data. By leveraging iterative mirror descent and importance sampling techniques, Off-MMD estimates the mean-field distribution from static datasets without relying on simulation or environment dynamics. Additionally, we incorporate techniques from offline reinforcement learning to address common issues like Q-value overestimation, ensuring robust policy learning even with limited data coverage. Our algorithm scales to complex environments and demonstrates strong performance on benchmark tasks like crowd exploration or navigation, highlighting its applicability to real-world multi-agent systems where online experimentation is infeasible. We empirically demonstrate the robustness of Off-MMD to low-quality datasets and conduct experiments to investigate its sensitivity to hyperparameter choices.

comment: Submitted to AAMAS

★ TranSPORTmer: A Holistic Approach to Trajectory Understanding in Multi-Agent Sports ACCV 2024

Guillem Capellera, Luis Ferraz, Antonio Rubio, Antonio Agudo, Francesc Moreno-Noguer

Understanding trajectories in multi-agent scenarios requires addressing various tasks, including predicting future movements, imputing missing observations, inferring the status of unseen agents, and classifying different global states. Traditional data-driven approaches often handle these tasks separately with specialized models. We introduce TranSPORTmer, a unified transformer-based framework capable of addressing all these tasks, showcasing its application to the intricate dynamics of multi-agent sports scenarios like soccer and basketball. Using Set Attention Blocks, TranSPORTmer effectively captures temporal dynamics and social interactions in an equivariant manner. The model's tasks are guided by an input mask that conceals missing or yet-to-be-predicted observations. Additionally, we introduce a CLS extra agent to classify states along soccer trajectories, including passes, possessions, uncontrolled states, and out-of-play intervals, contributing to an enhancement in modeling trajectories. Evaluations on soccer and basketball datasets show that TranSPORTmer outperforms state-of-the-art task-specific models in player forecasting, player forecasting-imputation, ball inference, and ball imputation. https://youtu.be/8VtSRm8oGoE

comment: Accepted to ACCV 2024

★ Markov Potential Game with Final-time Reach-Avoid Objectives

Sarah H. Q. Li, Abraham P. Vinod

We formulate a Markov potential game with final-time reach-avoid objectives by integrating potential game theory with stochastic reach-avoid control. Our focus is on multi-player trajectory planning where players maximize the same multi-player reach-avoid objective: the probability of all participants reaching their designated target states by a specified time, while avoiding collisions with one another. Existing approaches require centralized computation of actions via a global policy, which may have prohibitively expensive communication costs. Instead, we focus on approximations of the global policy via local state feedback policies. First, we adapt the recursive single player reach-avoid value iteration to the multi-player framework with local policies, and show that the same recursion holds on the joint state space. To find each player's optimal local policy, the multi-player reach-avoid value function is projected from the joint state to the local state using the other players' occupancy measures. Then, we propose an iterative best response scheme for the multi-player value iteration to converge to a pure Nash equilibrium. We demonstrate the utility of our approach in finding collision-free policies for multi-player motion planning in simulation.

comment: 8 pages, 2 figures

★ Bridging Swarm Intelligence and Reinforcement Learning

Karthik Soma, Yann Bouteiller, Heiko Hamann, Giovanni Beltrame

Swarm intelligence (SI) explores how large groups of simple individuals (e.g., insects, fish, birds) collaborate to produce complex behaviors, exemplifying that the whole is greater than the sum of its parts. A fundamental task in SI is Collective Decision-Making (CDM), where a group selects the best option among several alternatives, such as choosing an optimal foraging site. In this work, we demonstrate a theoretical and empirical equivalence between CDM and single-agent reinforcement learning (RL) in multi-armed bandit problems, utilizing concepts from opinion dynamics, evolutionary game theory, and RL. This equivalence bridges the gap between SI and RL and leads us to introduce a novel abstract RL update rule called Maynard-Cross Learning. Additionally, it provides a new population-based perspective on common RL practices like learning rate adjustment and batching. Our findings enable cross-disciplinary fertilization between RL and SI, allowing techniques from one field to enhance the understanding and methodologies of the other.

♻ ★ IBGP: Imperfect Byzantine Generals Problem for Zero-Shot Robustness in Communicative Multi-Agent Systems

Yihuan Mao, Yipeng Kang, Peilun Li, Ning Zhang, Wei Xu, Chongjie Zhang

As large language model (LLM) agents increasingly integrate into our infrastructure, their robust coordination and message synchronization become vital. The Byzantine Generals Problem (BGP) is a critical model for constructing resilient multi-agent systems (MAS) under adversarial attacks. It describes a scenario where malicious agents with unknown identities exist in the system-situations that, in our context, could result from LLM agents' hallucinations or external attacks. In BGP, the objective of the entire system is to reach a consensus on the action to be taken. Traditional BGP requires global consensus among all agents; however, in practical scenarios, global consensus is not always necessary and can even be inefficient. Therefore, there is a pressing need to explore a refined version of BGP that aligns with the local coordination patterns observed in MAS. We refer to this refined version as Imperfect BGP (IBGP) in our research, aiming to address this discrepancy. To tackle this issue, we propose a framework that leverages consensus protocols within general MAS settings, providing provable resilience against communication attacks and adaptability to changing environments, as validated by empirical results. Additionally, we present a case study in a sensor network environment to illustrate the practical application of our protocol.

♻ ★ Improve Value Estimation of Q Function and Reshape Reward with Monte Carlo Tree Search

Jiamian Li

Reinforcement learning has achieved remarkable success in perfect information games such as Go and Atari, enabling agents to compete at the highest levels against human players. However, research in reinforcement learning for imperfect information games has been relatively limited due to the more complex game structures and randomness. Traditional methods face challenges in training and improving performance in imperfect information games due to issues like inaccurate Q value estimation and reward sparsity. In this paper, we focus on Uno, an imperfect information game, and aim to address these problems by reducing Q value overestimation and reshaping reward function. We propose a novel algorithm that utilizes Monte Carlo Tree Search to average the value estimations in Q function. Even though we choose Double Deep Q Learning as the foundational framework in this paper, our method can be generalized and used in any algorithm which needs Q value estimation, such as the Actor-Critic. Additionally, we employ Monte Carlo Tree Search to reshape the reward structure in the game environment. We compare our algorithm with several traditional methods applied to games such as Double Deep Q Learning, Deep Monte Carlo and Neural Fictitious Self Play, and the experiments demonstrate that our algorithm consistently outperforms these approaches, especially as the number of players in Uno increases, indicating a higher level of difficulty.

♻ ★ Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy

Zhenyu Guan, Xiangyu Kong, Fangwei Zhong, Yizhou Wang

Diplomacy is one of the most sophisticated activities in human society, involving complex interactions among multiple parties that require skills in social reasoning, negotiation, and long-term strategic planning. Previous AI agents have demonstrated their ability to handle multi-step games and large action spaces in multi-agent tasks. However, diplomacy involves a staggering magnitude of decision spaces, especially considering the negotiation stage required. While recent agents based on large language models (LLMs) have shown potential in various applications, they still struggle with extended planning periods in complex multi-agent settings. Leveraging recent technologies for LLM-based agents, we aim to explore AI's potential to create a human-like agent capable of executing comprehensive multi-agent missions by integrating three fundamental capabilities: 1) strategic planning with memory and reflection; 2) goal-oriented negotiation with social reasoning; and 3) augmenting memory through self-play games for self-evolution without human in the loop.

♻ ★ Bearing-Distance Based Flocking with Zone-Based Interactions

Hossein B. Jond

This paper presents a novel zone-based flocking control approach suitable for dynamic multi-agent systems (MAS). Inspired by Reynolds behavioral rules for $boids$, flocking behavioral rules with the zones of repulsion, conflict, attraction, and surveillance are introduced. For each agent, using only bearing and distance measurements, behavioral deviation vectors quantify the deviations from the local separation, local and global flock velocity alignment, local cohesion, obstacle avoidance and boundary conditions, and strategic separation for avoiding alien agents. The control strategy uses the local perception-based behavioral deviation vectors to guide each agent's motion. Additionally, the control strategy incorporates a directionally-aware obstacle avoidance mechanism that prioritizes obstacles in the agent's forward path. Simulation results validate the effectiveness of this approach in creating flexible, adaptable, and scalable flocking behavior.

♻ ★ On the limits of agency in agent-based models

Ayush Chopra, Shashank Kumar, Nurullah Giray-Kuru, Ramesh Raskar, Arnau Quera-Bofarull

Agent-based modeling (ABM) seeks to understand the behavior of complex systems by simulating a collection of agents that act and interact within an environment. Their practical utility requires capturing realistic environment dynamics and adaptive agent behavior while efficiently simulating million-size populations. Recent advancements in large language models (LLMs) present an opportunity to enhance ABMs by using LLMs as agents with further potential to capture adaptive behavior. However, the computational infeasibility of using LLMs for large populations has hindered their widespread adoption. In this paper, we introduce AgentTorch -- a framework that scales ABMs to millions of agents while capturing high-resolution agent behavior using LLMs. We benchmark the utility of LLMs as ABM agents, exploring the trade-off between simulation scale and individual agency. Using the COVID-19 pandemic as a case study, we demonstrate how AgentTorch can simulate 8.4 million agents representing New York City, capturing the impact of isolation and employment behavior on health and economic outcomes. We compare the performance of different agent architectures based on heuristic and LLM agents in predicting disease waves and unemployment rates. Furthermore, we showcase AgentTorch's capabilities for retrospective, counterfactual, and prospective analyses, highlighting how adaptive agent behavior can help overcome the limitations of historical data in policy design. AgentTorch is an open-source project actively being used for policy-making and scientific discovery around the world. The framework is available here: github.com/AgentTorch/AgentTorch.

comment: 19 pages, 5 appendices, 5 figures

★ Effective Finite Time Stability Control for Human-Machine Shared Vehicle Following System

Zihan Wang, Mengran Li, Ronghui Zhang, Jing Zhao, Chuan Hu, Xiaolei Ma, Zhijun Qiu

With the development of intelligent connected vehicle technology, human-machine shared control has gained popularity in vehicle following due to its effectiveness in driver assistance. However, traditional vehicle following systems struggle to maintain stability when driver reaction time fluctuates, as these variations require different levels of system intervention. To address this issue, the proposed human-machine shared vehicle following assistance system (HM-VFAS) integrates driver outputs under various states with the assistance system. The system employs an intelligent driver model that accounts for reaction time delays, simulating time-varying driver outputs. A control authority allocation strategy is designed to dynamically adjust the level of intervention based on real-time driver state assessment. To handle instability from driver authority switching, the proposed solution includes a two-layer adaptive finite time sliding mode controller (A-FTSMC). The first layer is an integral sliding mode adaptive controller that ensures robustness by compensating for uncertainties in the driver output. The second layer is a fast non-singular terminal sliding mode controller designed to accelerate convergence for rapid stabilization. Using real driver videos as inputs, the performance of the HM-VFAS was evaluated. Results show that the proposed control strategy maintains a safe distance under time-varying driver states, with the actual acceleration error relative to the target acceleration maintained within 0.5m/s~2 and the maximum acceleration error reduced by 1.2m/s~2. Compared to traditional controllers, the A-FTSMC controller offers faster convergence and less vibration, reducing the stabilization time by 27.3%.

★ Role of hydrogen in decarbonizing China's electricity and hard-to-abate sectors

Haozhe Yang, Gang He, Eric Masanet, Ranjit Deshmukh

Green hydrogen has the potential to address two pressing problems in a zero-carbon energy system: balancing seasonal variability of solar and wind in the electricity sector, and replacing fossil fuels in hard-to-abate sectors. However, the previous research only separately modeled the electricity and hard-to-abate sectors, which is unable to capture how the interaction between the two sectors influences the energy system cost. In this study, focusing on China, we deploy an electricity system planning model to examine the cost implications of green hydrogen to fully decarbonize the electricity system and hard-to-abate sectors. Our results reveal that green hydrogen enables a 17% reduction in the levelized cost of a zero-carbon electricity system relative to that without hydrogen. However, cost savings hinge on the availability of underground hydrogen storage capacities and electric transmission expansion. More importantly, coupling hydrogen infrastructure in the electricity and hard-to-abate sectors not only reduces energy costs compared to a decoupled energy system but also makes green hydrogen cost-competitive compared to fossil fuel-based gray and blue hydrogen in China.

comment: 4 figures, 33 pages

★ Reconfigurable Hydrostatics: Toward Multifunctional and Powerful Wearable Robotics

Jeff Denis, Frederic Laberge, Jean-Sebastien Plante, Alexandre Girard

Wearable and locomotive robot designers face multiple challenges when choosing actuation. Traditional fully actuated designs using electric motors are multifunctional but oversized and inefficient for bearing conservative loads and for being backdrivable. Alternatively, quasi-passive and underactuated designs reduce the size of motorization and energy storage, but are often designed for specific tasks. Designers of versatile and stronger wearable robots will face these challenges unless future actuators become very torque-dense, backdrivable and efficient. This paper explores a design paradigm for addressing this issue: reconfigurable hydrostatics. We show that a hydrostatic actuator can integrate a passive force mechanism and a sharing mechanism in the fluid domain and still be multifunctional. First, an analytical study compares how these two mechanisms can relax the motorization requirements in the context of a load-bearing exoskeleton. Then, the hydrostatic concept integrating these two mechanisms using hydraulic components is presented. A case study analysis shows the mass/efficiency/inertia benefits of the concept over a fully actuated one. Then, the feasibility of the concept is partially validated with a proof-of-concept that actuates the knees of an exoskeleton. The experiments show that it can track the vertical ground reaction force (GRF) profiles of walking, running, squatting, and jumping, and that the energy consumption is 6x lower. The transient force behaviors due to switching from one leg to the other are also analyzed along with some mitigation to improve them.

★ Identifiable Representation and Model Learning for Latent Dynamic Systems

Congxi Zhang, Yongchun Xie

Learning identifiable representations and models from low-level observations is useful for an intelligent spacecraft to reliability finish downstream tasks. For temporal observations, to ensure that the data generating process is provably inverted, most existing works either assume the noise variables in the dynamic mechanisms are (conditionally) independent, or require interventions which can directly affect each latent variable. However, in practice, the relationship between the exogenous inputs/interventions and the latent variables may follow some complex deterministic mechanisms. In this work, we study the problem of identifiable representation and model learning for latent dynamic systems. The key idea is that we use an inductive bias inspired by controllable canonical forms, which is invariant, sparse, and input dependent by definition. We prove that, for linear or affine nonlinear latent dynamic systems, it is possible to identify the representations up to scaling and determine the models up to some simple transformations. The results have potential to provide some theoretical guarantees for developing more trustworthy decision-making and control methods for intelligent spacecrafts.

★ An iteration-free approach to excitation harmonization

Patrick Hippold, Gleb Kleyman, Lukas Woiwode, Tong Wei, Florian Müller, Christoph Schwingshackl, Maren Scheel, Sebastian Tatzko, Malte Krack

Sinusoidal excitation is particularly popular for testing structures in the nonlinear regime. Due to the nonlinear behavior and the inevitable feedback of the structure on the exciter, higher harmonics in the applied excitation are generated. This is undesired, because the acquired response may deviate substantially from that of the structure under purely sinusoidal excitation, in particular if one of the higher harmonics engages into resonance. We present a new approach to suppress those higher excitation harmonics and thus the unwanted exciter-structure interaction: Higher harmonics are added to the voltage input to the shaker whose Fourier coefficients are adjusted via feedback control until the excitation is purely sinusoidal. The stability of this method is analyzed for a simplified model; the resulting closed-form expressions are useful, among others, to select an appropriate exciter configuration, including the drive point. A practical procedure for the control design is suggested. The proposed method is validated in virtual and real experiments of internally resonant structures, in the two common configurations of force excitation via a stinger and base excitation. Excellent performance is achieved already when using the same control gains for all harmonics, throughout the tested range of amplitudes and frequencies, even in the strongly nonlinear regime. Compared to the iterative state of the art, it is found that the proposed method is simpler to implement, enables faster testing and it is easy to achieve a lower harmonic distortion.

★ e-Values for Real-Time Residential Electricity Demand Forecast Model Selection

Fabian Backhaus, Karoline Brucke, Peter Ruckdeschel, Sunke Schlüters

With the growing number of forecasting techniques and the increasing significance of forecast-based operation - particularly in the rapidly evolving energy sector - selecting the most effective forecasting model has become a critical task. Given the dynamic nature of energy forecasting, it is highly advantageous to assess the superiority of forecasting models not only retrospectively but continuously in real-time as new data and evidence becomes available, while simultaneously providing strong probabilistic guarantees for these decisions. In this work, we show that this can be achieved through the mathematical concept of e-values, which has recently gained massive attention in the field of statistics. It allows for unified construction principles for powerful tests and accurate statistical decisions, which can be evaluated at any chosen time points while maintaining an overall probabilistic error control. We extend the use of e-values by developing a simple persistence approach that dynamically combines input forecasts to generate new fused predictions. To demonstrate the performance of our method we apply it to electricity demand forecasts based on different artificial intelligence based models. Our results indicate that e-values are able to improve the accuracy and reliability of forecasts in a dynamic environment, offering a valuable tool for real-time decision-making in the energy sector.

comment: 25 pages, 9 figures

★ Fiber Activation by Bipolar Stimulation in Deep Brain Stimulation: A Patient Case Study

Anna Franziska Frigge, Elena Jiltsova, Fredrik Olsson, Dag Nyholm, Alexander Medvedev

Deep Brain Stimulation (DBS) is a therapy widely used for treating the symptoms of neurological disorders. Electrical pulses are chronically delivered in DBS to a disease-specific brain target via a surgically implanted electrode. The stimulating contact configuration, stimulation polarity, as well as amplitude, frequency, and pulse width of the DBS pulse sequence are utilized to optimize the therapeutic effect. In this paper, the utility of therapy individualization by means of patient-specific mathematical modeling is investigated with respect to a specific case of a patient diagnosed with Essential Tremor (ET). Two computational models are compared in their ability to elucidate the impact of DBS stimulation on the dentato-rubrothalamic tract: (i) a conventional model of Volume of Tissue Activated (VTA) and (ii) a well-established neural fiber activation modeling framework known as OSS-DBS. The simulation results are compared with tremor measured in the patient under different DBS settings using a smartphone application. The findings of the study highlight that temporally static VTA models do not adequately describe the differences in the outcomes of bipolar stimulation settings with switched polarity, whereas neural fiber activation models hold potential in this regard. However, it is noted that neither of the investigated models fully accounts for the measured symptom pattern, particularly regarding a bilateral effect produced by unilateral stimulation.

comment: 7 pages, 5 figures, 5 tables

★ Time-to-Lie: Identifying Industrial Control System Honeypots Using the Internet Control Message Protocol

Jacob Williams, Matthew Edwards, Joseph Gardiner

The convergence of information and operational technology networks has created previously unforeseen security issues. To address these issues, both researchers and practitioners have integrated threat intelligence methods into the security operations of converged networks, with some of the most valuable tools being honeypots that imitate industrial control systems (ICS). However, the development and deployment of such honeypots is a process rich with pitfalls, which can lead to undiagnosed weaknesses in the threat intelligence being gathered. This paper presents a side-channel method of covertly identifying ICS honeypots using the time-to-live (TTL) values of target devices. We show that many ICS honeypots can be readily identified, via minimal interactions, using only basic networking tools. In a study of over 8,000 devices presenting as ICS systems, we detail how our method compares to an existing honeypot detection approach, and outline what our methodology reveals about the current population of live ICS honeypots. In demonstrating our method, this study aims to raise awareness of the viability of the TTL heuristic and the prevalence of its misconfiguration despite its presence in literature.

comment: 11 pages, 2 listings, 5 tables, 6 figures

★ Deoxys: A Causal Inference Engine for Unhealthy Node Mitigation in Large-scale Cloud Infrastructure

Chaoyun Zhang, Randolph Yao, Si Qin, Ze Li, Shekhar Agrawal, Binit R. Mishra, Tri Tran, Minghua Ma, Qingwei Lin, Murali Chintalapati, Dongmei Zhang

The presence of unhealthy nodes in cloud infrastructure signals the potential failure of machines, which can significantly impact the availability and reliability of cloud services, resulting in negative customer experiences. Effectively addressing unhealthy node mitigation is therefore vital for sustaining cloud system performance. This paper introduces Deoxys, a causal inference engine tailored to recommending mitigation actions for unhealthy node in cloud systems to minimize virtual machine downtime and interruptions during unhealthy events. It employs double machine learning combined with causal forest to produce precise and reliable mitigation recommendations based solely on limited observational data collected from the historical unhealthy events. To enhance the causal inference model, Deoxys further incorporates a policy fallback mechanism based on model uncertainty and action overriding mechanisms to (i) improve the reliability of the system, and (ii) strike a good tradeoff between downtime reduction and resource utilization, thereby enhancing the overall system performance. After deploying Deoxys in a large-scale cloud infrastructure at Microsoft, our observations demonstrate that Deoxys significantly reduces average VM downtime by 53% compared to a legacy policy, while leading to 49.5% lower VM interruption rate. This substantial improvement enhances the reliability and stability of cloud platforms, resulting in a seamless customer experience.

★ Markov Potential Game with Final-time Reach-Avoid Objectives

Sarah H. Q. Li, Abraham P. Vinod

We formulate a Markov potential game with final-time reach-avoid objectives by integrating potential game theory with stochastic reach-avoid control. Our focus is on multi-player trajectory planning where players maximize the same multi-player reach-avoid objective: the probability of all participants reaching their designated target states by a specified time, while avoiding collisions with one another. Existing approaches require centralized computation of actions via a global policy, which may have prohibitively expensive communication costs. Instead, we focus on approximations of the global policy via local state feedback policies. First, we adapt the recursive single player reach-avoid value iteration to the multi-player framework with local policies, and show that the same recursion holds on the joint state space. To find each player's optimal local policy, the multi-player reach-avoid value function is projected from the joint state to the local state using the other players' occupancy measures. Then, we propose an iterative best response scheme for the multi-player value iteration to converge to a pure Nash equilibrium. We demonstrate the utility of our approach in finding collision-free policies for multi-player motion planning in simulation.

comment: 8 pages, 2 figures

★ Accelerating soft-constrained MPC for linear systems through online constraint removal

S. A. N. Nouwens, M. M. Paulides, W. P. M. H. Heemels

Optimization-based controllers, such as Model Predictive Control (MPC), have attracted significant research interest due to their intuitive concept, constraint handling capabilities, and natural application to multi-input multi-output systems. However, the computational complexity of solving a receding horizon problem at each time step remains a challenge for the deployment of MPC. This is particularly the case for systems constrained by many inequalities. Recently, we introduced the concept of constraint-adaptive MPC (ca-MPC) to address this challenge for linear systems with hard constraints. In ca-MPC, at each time step, a subset of the constraints is removed from the optimization problem, thereby accelerating the optimization procedure, while resulting in identical closed-loop behavior. The present paper extends this framework to soft-constrained MPC by detecting and removing constraints based on sub-optimal predicted input sequences, which is rather easy for soft-constrained MPC due to the receding horizon principle and the inclusion of slack variables. We will translate these new ideas explicitly to an offset-free output tracking problem. The effectiveness of these ideas is demonstrated on a two-dimensional thermal transport model, showing a three order of magnitude improvement in online computational time of the MPC scheme.

comment: 6 pages, 5 figures, CDC 2023 conference

★ Approximate Kalman filtering for large-scale systems with an application to hyperthermia cancer treatments

S. A. N. Nouwens, M. M. Paulides, W. P. M. H. Heemels

Accurate state estimates are required for increasingly complex systems, to enable, for example, feedback control. However, available state estimation schemes are not necessarily real-time feasible for certain large-scale systems. Therefore, we develop in this paper, a real-time feasible state-estimation scheme for a class of large-scale systems that approximates the steady state Kalman filter. In particular, we focus on systems where the state-vector is the result of discretizing the spatial domain, as typically seen in Partial Differential Equations. In such cases, the correlation between states in the state-vector often have an intuitive interpretation on the spatial domain, which can be exploited to obtain a significant reduction in computational complexity, while still providing accurate state estimates. We illustrate these strengths of our method through a hyperthermia cancer treatment case study. The results of the case study show significant improvements in the computation time, while simultaneously obtaining good state estimates, when compared to Ensemble Kalman filters and Kalman filters using reduced-order models.

comment: 6 pages, 6 figures, CDC 2022 conference

★ Constraint Removal for MPC with Performance Preservation and a Hyperthermia Cancer Treatment Case Study

S. A. N. Nouwens, B. de Jager, M. M. Paulides, W. P. M. H. Heemels

Model predictive control (MPC) is an optimization-based control strategy with broad industrial adoption. Unfortunately, the required computation time to solve the receding-horizon MPC optimization problem can become prohibitively large for many applications with a large number of state constraints. This large number of state constraints can, for instance, originate from spatially discretizing a partial differential equation of which the solution has to satisfy constraints over the full spatial domain. This is particularly the case in MPC for RF-based hyperthermia cancer treatments, which forms a strong motivation for this study. To address this problem, we propose a novel constraint-adaptive MPC framework for linear discrete-time systems. In this framework, we select at each time-step a subset of the state constraints that are included in the optimization problem, thereby reducing the online computational burden. Critically, our framework guarantees the same closed-loop performance, recursive feasibility, and constraint satisfaction properties as the original (non-reduced) MPC scheme. We achieve this result by efficiently exploiting reachable set computations and the MPC cost function. We will demonstrate our novel method using a hyperthermia cancer treatment case study showing a two-orders of magnitude improvement in computation time, with identical closed-loop performance as the original (non-reduced) MPC scheme.

comment: 6 pages, 3 figures, CDC 2021 conference

★ Exploiting Data Centres and Local Energy Communities Synergies for Market Participation

Ángel Paredes, Yihong Zhou, Chaimaa Essayeh, José A. Aguado, Thomas Morstyn

The evolving energy landscape has propelled energy communities to the forefront of modern energy management. However, existing research has yet to explore the potential synergies between data centres and energy communities, necessitating an assessment on their collective capabilities for cost efficiency, waste heat optimisation, and market participation. This paper presents a mixed integer linear programming model to assess the collaborative performance of energy communities, data centres and energy markets. The evaluation focuses on the efficient use of waste heat and the flexibility of job scheduling while minimising system energy costs and maintaining quality of service requirements for data centres. Our results, based on realistic profiles of an energy community and a data centre, showcase significant benefits of these synergies, with a 38% reduction in operating costs and an 87% decrease in heat demand.

comment: Submitted to IEEE PES ISGT Europe 2024

★ Risk-sensitive Affine Control Synthesis for Stationary LTI Systems

Yang Hu, Shahriar Talebi, Na Li

To address deviations from expected performance in stochastic systems, we propose a risk-sensitive control synthesis method to minimize certain risk measures over the limiting stationary distribution. Specifically, we extend Worst-case Conditional Value-at-Risk (W-CVaR) optimization for Linear Time-invariant (LTI) systems to handle nonzero-mean noise and affine controllers, using only the first and second moments of noise, which enhances robustness against model uncertainty. Highlighting the strong coupling between the linear and bias terms of the controller, we reformulate the synthesis problem as a Bilinear Matrix Inequality (BMI), and propose an alternating optimization algorithm with guaranteed convergence. Finally, we demonstrate the numerical performance of our approach in two representative settings, which shows that the proposed algorithm successfully synthesizes risk-sensitive controllers that outperform the na\"ive LQR baseline.

comment: 8 pages, 4 figures, 2 illustrations

★ Real-time Vehicle-to-Vehicle Communication Based Network Cooperative Control System through Distributed Database and Multimodal Perception: Demonstrated in Crossroads

Xinwen Zhu, Zihao Li, Yuxuan Jiang, Jiazhen Xu, Jie Wang, Xuyang Bai

The autonomous driving industry is rapidly advancing, with Vehicle-to-Vehicle (V2V) communication systems highlighting as a key component of enhanced road safety and traffic efficiency. This paper introduces a novel Real-time Vehicle-to-Vehicle Communication Based Network Cooperative Control System (VVCCS), designed to revolutionize macro-scope traffic planning and collision avoidance in autonomous driving. Implemented on Quanser Car (Qcar) hardware platform, our system integrates the distributed databases into individual autonomous vehicles and an optional central server. We also developed a comprehensive multi-modal perception system with multi-objective tracking and radar sensing. Through a demonstration within a physical crossroad environment, our system showcases its potential to be applied in congested and complex urban environments.

comment: ICICT 2024, 18 pages

★ LEADS: Lightweight Embedded Assisted Driving System

Tianhao Fu, Querobin Mascarenhas, Andrew Forti

With the rapid development of electric vehicles, formula races that face high school and university students have become more popular than ever as the threshold for design and manufacturing has been lowered. In many cases, we see teams inspired by or directly using toolkits and technologies inherited from standardized commercial vehicles. These architectures are usually overly complicated for amateur applications like the races. In order to improve the efficiency and simplify the development of instrumentation, control, and analysis systems, we propose LEADS (Lightweight Embedded Assisted Driving System), a dedicated solution for such scenarios.

comment: 12 pages

★ Electric Grid Topology and Admittance Estimation: Quantifying Phasor-based Measurement Requirements

Norak Rin, Iman Shames, Ian R. Petersen, Elizabeth L. Ratnam

In this paper, we quantify voltage and current phasor-based measurement requirements for the unique identification of the electric grid topology and admittance parameters. Our approach is underpinned by the concept of a rigidity matrix that has been extensively studied in graph rigidity theory. Specifically, we show that the rank of the rigidity matrix is the same as that of a voltage coefficient matrix in a corresponding electric power system. Accordingly, we show that there is a minimum number of measurements required to uniquely identify the admittance matrix and corresponding grid topology. By means of a numerical example on the IEEE 4-node radial network, we demonstrate that our approach is suitable for applications in electric power grids.

♻ ★ SPARC: Prediction-Based Safe Control for Coupled Controllable and Uncontrollable Agents with Conformal Predictions

Shuqi Wang, Shaoyuan Li, Xiang Yin

We investigate the problem of safe control synthesis for systems operating in environments with uncontrollable agents whose dynamics are unknown but coupled with those of the controlled system. This scenario naturally arises in various applications, such as autonomous driving and human-robot collaboration, where the behavior of uncontrollable agents, like pedestrians, cannot be directly controlled but is influenced by the actions of the autonomous vehicle or robot. In this paper, we present SPARC (Safe Prediction-Based Robust Controller for Coupled Agents), a novel framework designed to ensure safe control in the presence of coupled uncontrollable agents. SPARC leverages conformal prediction to quantify uncertainty in data-driven prediction of agent behavior. Particularly, we introduce a joint distribution-based approach to account for the coupled dynamics of the controlled system and uncontrollable agents. By integrating the control barrier function (CBF) technique, SPARC provides provable safety guarantees at a high confidence level. We illustrate our framework with a case study involving an autonomous driving scenario with walking pedestrians.

comment: It's not complete yet

♻ ★ Addressing Trust Issues for Vehicle to Grid in Distributed Power Grids Using Blockchains

Yunwang Chen, Xiang Lei, Linni Jian

While blockchain offers inherent security, trust issues among stakeholders in vehicle-to-grid (V2G) applications remain unresolved due to a lack of regulatory frameworks and standardization. Additionally, a tailored decentralized privacy-preserved coordination scheme for blockchain in V2G networks is needed to ensure user privacy and efficient energy transactions. This paper proposes a V2G trading and coordination scheme tailored to the decentralized nature of blockchain as well as the interests of stakeholders utilizing smart charging points (SCPs) and Stackelberg game model. Case studies using real-world data from Southern University of Science and Technology demonstrate the efficacy of proposed scheme in reducing EV charging costs and the potential for supporting auxiliary grid services.

comment: This paper has been accepted by The 14th International Conference on Power and Energy Systems (ICPES 2024)

♻ ★ Certifiably Robust Policies for Uncertain Parametric Environments

Yannik Schnitzer, Alessandro Abate, David Parker

We present a data-driven approach for producing policies that are provably robust across unknown stochastic environments. Existing approaches can learn models of a single environment as an interval Markov decision processes (IMDP) and produce a robust policy with a probably approximately correct (PAC) guarantee on its performance. However these are unable to reason about the impact of environmental parameters underlying the uncertainty. We propose a framework based on parametric Markov decision processes (MDPs) with unknown distributions over parameters. We learn and analyse IMDPs for a set of unknown sample environments induced by parameters. The key challenge is then to produce meaningful performance guarantees that combine the two layers of uncertainty: (1) multiple environments induced by parameters with an unknown distribution; (2) unknown induced environments which are approximated by IMDPs. We present a novel approach based on scenario optimisation that yields a single PAC guarantee quantifying the risk level for which a specified performance level can be assured in unseen environments, plus a means to trade-off risk and performance. We implement and evaluate our framework using multiple robust policy generation methods on a range of benchmarks. We show that our approach produces tight bounds on a policy's performance with high confidence.

♻ ★ Posterior Sampling-based Online Learning for Episodic POMDPs

Dengwang Tang, Dongze Ye, Rahul Jain, Ashutosh Nayyar, Pierluigi Nuzzo

Learning in POMDPs is known to be significantly harder than in MDPs. In this paper, we consider the online learning problem for episodic POMDPs with unknown transition and observation models. We propose a Posterior Sampling-based reinforcement learning algorithm for POMDPs (PS4POMDPs), which is much simpler and more implementable compared to state-of-the-art optimism-based online learning algorithms for POMDPs. We show that the Bayesian regret of the proposed algorithm scales as the square root of the number of episodes and is polynomial in the other parameters. In a general setting, the regret scales exponentially in the horizon length $H$, and we show that this is inevitable by providing a lower bound. However, when the POMDP is undercomplete and weakly revealing (a common assumption in the recent literature), we establish a polynomial Bayesian regret bound. We finally propose a posterior sampling algorithm for multi-agent POMDPs, and show it too has sublinear regret.

comment: 41 pages, 9 figures

♻ ★ Mitigating Information Asymmetry in Two-Stage Contracts with Non-Myopic Agents

Munther A. Dahleh, Thibaut Horel, M. Umar B. Niazi

We consider a Stackelberg game in which a principal (she) establishes a two-stage contract with a non-myopic agent (he) whose type is unknown. The contract takes the form of an incentive function mapping the agent's first-stage action to his second-stage incentive. While the first-stage action reveals the agent's type under truthful play, a non-myopic agent could benefit from portraying a false type in the first stage to obtain a larger incentive in the second stage. The challenge is thus for the principal to design the incentive function so as to induce truthful play. We show that this is only possible with a constant, non-reactive incentive functions when the type space is continuous, whereas it can be achieved with reactive functions for discrete types. Additionally, we show that introducing an adjustment mechanism that penalizes inconsistent behavior across both stages allows the principal to design more flexible incentive functions.

comment: To appear in the Proceedings of the 5th IFAC Workshop on Cyber-Physical Human Systems

★ Effective Finite Time Stability Control for Human-Machine Shared Vehicle Following System

Zihan Wang, Mengran Li, Ronghui Zhang, Jing Zhao, Chuan Hu, Xiaolei Ma, Zhijun Qiu

With the development of intelligent connected vehicle technology, human-machine shared control has gained popularity in vehicle following due to its effectiveness in driver assistance. However, traditional vehicle following systems struggle to maintain stability when driver reaction time fluctuates, as these variations require different levels of system intervention. To address this issue, the proposed human-machine shared vehicle following assistance system (HM-VFAS) integrates driver outputs under various states with the assistance system. The system employs an intelligent driver model that accounts for reaction time delays, simulating time-varying driver outputs. A control authority allocation strategy is designed to dynamically adjust the level of intervention based on real-time driver state assessment. To handle instability from driver authority switching, the proposed solution includes a two-layer adaptive finite time sliding mode controller (A-FTSMC). The first layer is an integral sliding mode adaptive controller that ensures robustness by compensating for uncertainties in the driver output. The second layer is a fast non-singular terminal sliding mode controller designed to accelerate convergence for rapid stabilization. Using real driver videos as inputs, the performance of the HM-VFAS was evaluated. Results show that the proposed control strategy maintains a safe distance under time-varying driver states, with the actual acceleration error relative to the target acceleration maintained within 0.5m/s~2 and the maximum acceleration error reduced by 1.2m/s~2. Compared to traditional controllers, the A-FTSMC controller offers faster convergence and less vibration, reducing the stabilization time by 27.3%.

★ Role of hydrogen in decarbonizing China's electricity and hard-to-abate sectors

Haozhe Yang, Gang He, Eric Masanet, Ranjit Deshmukh

Green hydrogen has the potential to address two pressing problems in a zero-carbon energy system: balancing seasonal variability of solar and wind in the electricity sector, and replacing fossil fuels in hard-to-abate sectors. However, the previous research only separately modeled the electricity and hard-to-abate sectors, which is unable to capture how the interaction between the two sectors influences the energy system cost. In this study, focusing on China, we deploy an electricity system planning model to examine the cost implications of green hydrogen to fully decarbonize the electricity system and hard-to-abate sectors. Our results reveal that green hydrogen enables a 17% reduction in the levelized cost of a zero-carbon electricity system relative to that without hydrogen. However, cost savings hinge on the availability of underground hydrogen storage capacities and electric transmission expansion. More importantly, coupling hydrogen infrastructure in the electricity and hard-to-abate sectors not only reduces energy costs compared to a decoupled energy system but also makes green hydrogen cost-competitive compared to fossil fuel-based gray and blue hydrogen in China.

comment: 4 figures, 33 pages

★ Reconfigurable Hydrostatics: Toward Multifunctional and Powerful Wearable Robotics

Jeff Denis, Frederic Laberge, Jean-Sebastien Plante, Alexandre Girard

Wearable and locomotive robot designers face multiple challenges when choosing actuation. Traditional fully actuated designs using electric motors are multifunctional but oversized and inefficient for bearing conservative loads and for being backdrivable. Alternatively, quasi-passive and underactuated designs reduce the size of motorization and energy storage, but are often designed for specific tasks. Designers of versatile and stronger wearable robots will face these challenges unless future actuators become very torque-dense, backdrivable and efficient. This paper explores a design paradigm for addressing this issue: reconfigurable hydrostatics. We show that a hydrostatic actuator can integrate a passive force mechanism and a sharing mechanism in the fluid domain and still be multifunctional. First, an analytical study compares how these two mechanisms can relax the motorization requirements in the context of a load-bearing exoskeleton. Then, the hydrostatic concept integrating these two mechanisms using hydraulic components is presented. A case study analysis shows the mass/efficiency/inertia benefits of the concept over a fully actuated one. Then, the feasibility of the concept is partially validated with a proof-of-concept that actuates the knees of an exoskeleton. The experiments show that it can track the vertical ground reaction force (GRF) profiles of walking, running, squatting, and jumping, and that the energy consumption is 6x lower. The transient force behaviors due to switching from one leg to the other are also analyzed along with some mitigation to improve them.

★ Identifiable Representation and Model Learning for Latent Dynamic Systems

Congxi Zhang, Yongchun Xie

Learning identifiable representations and models from low-level observations is useful for an intelligent spacecraft to reliability finish downstream tasks. For temporal observations, to ensure that the data generating process is provably inverted, most existing works either assume the noise variables in the dynamic mechanisms are (conditionally) independent, or require interventions which can directly affect each latent variable. However, in practice, the relationship between the exogenous inputs/interventions and the latent variables may follow some complex deterministic mechanisms. In this work, we study the problem of identifiable representation and model learning for latent dynamic systems. The key idea is that we use an inductive bias inspired by controllable canonical forms, which is invariant, sparse, and input dependent by definition. We prove that, for linear or affine nonlinear latent dynamic systems, it is possible to identify the representations up to scaling and determine the models up to some simple transformations. The results have potential to provide some theoretical guarantees for developing more trustworthy decision-making and control methods for intelligent spacecrafts.

★ An iteration-free approach to excitation harmonization

Patrick Hippold, Gleb Kleyman, Lukas Woiwode, Tong Wei, Florian Müller, Christoph Schwingshackl, Maren Scheel, Sebastian Tatzko, Malte Krack

Sinusoidal excitation is particularly popular for testing structures in the nonlinear regime. Due to the nonlinear behavior and the inevitable feedback of the structure on the exciter, higher harmonics in the applied excitation are generated. This is undesired, because the acquired response may deviate substantially from that of the structure under purely sinusoidal excitation, in particular if one of the higher harmonics engages into resonance. We present a new approach to suppress those higher excitation harmonics and thus the unwanted exciter-structure interaction: Higher harmonics are added to the voltage input to the shaker whose Fourier coefficients are adjusted via feedback control until the excitation is purely sinusoidal. The stability of this method is analyzed for a simplified model; the resulting closed-form expressions are useful, among others, to select an appropriate exciter configuration, including the drive point. A practical procedure for the control design is suggested. The proposed method is validated in virtual and real experiments of internally resonant structures, in the two common configurations of force excitation via a stinger and base excitation. Excellent performance is achieved already when using the same control gains for all harmonics, throughout the tested range of amplitudes and frequencies, even in the strongly nonlinear regime. Compared to the iterative state of the art, it is found that the proposed method is simpler to implement, enables faster testing and it is easy to achieve a lower harmonic distortion.

★ e-Values for Real-Time Residential Electricity Demand Forecast Model Selection

Fabian Backhaus, Karoline Brucke, Peter Ruckdeschel, Sunke Schlüters

With the growing number of forecasting techniques and the increasing significance of forecast-based operation - particularly in the rapidly evolving energy sector - selecting the most effective forecasting model has become a critical task. Given the dynamic nature of energy forecasting, it is highly advantageous to assess the superiority of forecasting models not only retrospectively but continuously in real-time as new data and evidence becomes available, while simultaneously providing strong probabilistic guarantees for these decisions. In this work, we show that this can be achieved through the mathematical concept of e-values, which has recently gained massive attention in the field of statistics. It allows for unified construction principles for powerful tests and accurate statistical decisions, which can be evaluated at any chosen time points while maintaining an overall probabilistic error control. We extend the use of e-values by developing a simple persistence approach that dynamically combines input forecasts to generate new fused predictions. To demonstrate the performance of our method we apply it to electricity demand forecasts based on different artificial intelligence based models. Our results indicate that e-values are able to improve the accuracy and reliability of forecasts in a dynamic environment, offering a valuable tool for real-time decision-making in the energy sector.

comment: 25 pages, 9 figures

★ Fiber Activation by Bipolar Stimulation in Deep Brain Stimulation: A Patient Case Study

Anna Franziska Frigge, Elena Jiltsova, Fredrik Olsson, Dag Nyholm, Alexander Medvedev

Deep Brain Stimulation (DBS) is a therapy widely used for treating the symptoms of neurological disorders. Electrical pulses are chronically delivered in DBS to a disease-specific brain target via a surgically implanted electrode. The stimulating contact configuration, stimulation polarity, as well as amplitude, frequency, and pulse width of the DBS pulse sequence are utilized to optimize the therapeutic effect. In this paper, the utility of therapy individualization by means of patient-specific mathematical modeling is investigated with respect to a specific case of a patient diagnosed with Essential Tremor (ET). Two computational models are compared in their ability to elucidate the impact of DBS stimulation on the dentato-rubrothalamic tract: (i) a conventional model of Volume of Tissue Activated (VTA) and (ii) a well-established neural fiber activation modeling framework known as OSS-DBS. The simulation results are compared with tremor measured in the patient under different DBS settings using a smartphone application. The findings of the study highlight that temporally static VTA models do not adequately describe the differences in the outcomes of bipolar stimulation settings with switched polarity, whereas neural fiber activation models hold potential in this regard. However, it is noted that neither of the investigated models fully accounts for the measured symptom pattern, particularly regarding a bilateral effect produced by unilateral stimulation.

comment: 7 pages, 5 figures, 5 tables

★ Time-to-Lie: Identifying Industrial Control System Honeypots Using the Internet Control Message Protocol

Jacob Williams, Matthew Edwards, Joseph Gardiner

The convergence of information and operational technology networks has created previously unforeseen security issues. To address these issues, both researchers and practitioners have integrated threat intelligence methods into the security operations of converged networks, with some of the most valuable tools being honeypots that imitate industrial control systems (ICS). However, the development and deployment of such honeypots is a process rich with pitfalls, which can lead to undiagnosed weaknesses in the threat intelligence being gathered. This paper presents a side-channel method of covertly identifying ICS honeypots using the time-to-live (TTL) values of target devices. We show that many ICS honeypots can be readily identified, via minimal interactions, using only basic networking tools. In a study of over 8,000 devices presenting as ICS systems, we detail how our method compares to an existing honeypot detection approach, and outline what our methodology reveals about the current population of live ICS honeypots. In demonstrating our method, this study aims to raise awareness of the viability of the TTL heuristic and the prevalence of its misconfiguration despite its presence in literature.

comment: 11 pages, 2 listings, 5 tables, 6 figures

★ Deoxys: A Causal Inference Engine for Unhealthy Node Mitigation in Large-scale Cloud Infrastructure

Chaoyun Zhang, Randolph Yao, Si Qin, Ze Li, Shekhar Agrawal, Binit R. Mishra, Tri Tran, Minghua Ma, Qingwei Lin, Murali Chintalapati, Dongmei Zhang

The presence of unhealthy nodes in cloud infrastructure signals the potential failure of machines, which can significantly impact the availability and reliability of cloud services, resulting in negative customer experiences. Effectively addressing unhealthy node mitigation is therefore vital for sustaining cloud system performance. This paper introduces Deoxys, a causal inference engine tailored to recommending mitigation actions for unhealthy node in cloud systems to minimize virtual machine downtime and interruptions during unhealthy events. It employs double machine learning combined with causal forest to produce precise and reliable mitigation recommendations based solely on limited observational data collected from the historical unhealthy events. To enhance the causal inference model, Deoxys further incorporates a policy fallback mechanism based on model uncertainty and action overriding mechanisms to (i) improve the reliability of the system, and (ii) strike a good tradeoff between downtime reduction and resource utilization, thereby enhancing the overall system performance. After deploying Deoxys in a large-scale cloud infrastructure at Microsoft, our observations demonstrate that Deoxys significantly reduces average VM downtime by 53% compared to a legacy policy, while leading to 49.5% lower VM interruption rate. This substantial improvement enhances the reliability and stability of cloud platforms, resulting in a seamless customer experience.

★ Markov Potential Game with Final-time Reach-Avoid Objectives

Sarah H. Q. Li, Abraham P. Vinod

We formulate a Markov potential game with final-time reach-avoid objectives by integrating potential game theory with stochastic reach-avoid control. Our focus is on multi-player trajectory planning where players maximize the same multi-player reach-avoid objective: the probability of all participants reaching their designated target states by a specified time, while avoiding collisions with one another. Existing approaches require centralized computation of actions via a global policy, which may have prohibitively expensive communication costs. Instead, we focus on approximations of the global policy via local state feedback policies. First, we adapt the recursive single player reach-avoid value iteration to the multi-player framework with local policies, and show that the same recursion holds on the joint state space. To find each player's optimal local policy, the multi-player reach-avoid value function is projected from the joint state to the local state using the other players' occupancy measures. Then, we propose an iterative best response scheme for the multi-player value iteration to converge to a pure Nash equilibrium. We demonstrate the utility of our approach in finding collision-free policies for multi-player motion planning in simulation.

comment: 8 pages, 2 figures

★ Accelerating soft-constrained MPC for linear systems through online constraint removal

S. A. N. Nouwens, M. M. Paulides, W. P. M. H. Heemels

Optimization-based controllers, such as Model Predictive Control (MPC), have attracted significant research interest due to their intuitive concept, constraint handling capabilities, and natural application to multi-input multi-output systems. However, the computational complexity of solving a receding horizon problem at each time step remains a challenge for the deployment of MPC. This is particularly the case for systems constrained by many inequalities. Recently, we introduced the concept of constraint-adaptive MPC (ca-MPC) to address this challenge for linear systems with hard constraints. In ca-MPC, at each time step, a subset of the constraints is removed from the optimization problem, thereby accelerating the optimization procedure, while resulting in identical closed-loop behavior. The present paper extends this framework to soft-constrained MPC by detecting and removing constraints based on sub-optimal predicted input sequences, which is rather easy for soft-constrained MPC due to the receding horizon principle and the inclusion of slack variables. We will translate these new ideas explicitly to an offset-free output tracking problem. The effectiveness of these ideas is demonstrated on a two-dimensional thermal transport model, showing a three order of magnitude improvement in online computational time of the MPC scheme.

comment: 6 pages, 5 figures, CDC 2023 conference

★ Approximate Kalman filtering for large-scale systems with an application to hyperthermia cancer treatments

S. A. N. Nouwens, M. M. Paulides, W. P. M. H. Heemels

Accurate state estimates are required for increasingly complex systems, to enable, for example, feedback control. However, available state estimation schemes are not necessarily real-time feasible for certain large-scale systems. Therefore, we develop in this paper, a real-time feasible state-estimation scheme for a class of large-scale systems that approximates the steady state Kalman filter. In particular, we focus on systems where the state-vector is the result of discretizing the spatial domain, as typically seen in Partial Differential Equations. In such cases, the correlation between states in the state-vector often have an intuitive interpretation on the spatial domain, which can be exploited to obtain a significant reduction in computational complexity, while still providing accurate state estimates. We illustrate these strengths of our method through a hyperthermia cancer treatment case study. The results of the case study show significant improvements in the computation time, while simultaneously obtaining good state estimates, when compared to Ensemble Kalman filters and Kalman filters using reduced-order models.

comment: 6 pages, 6 figures, CDC 2022 conference

★ Constraint Removal for MPC with Performance Preservation and a Hyperthermia Cancer Treatment Case Study

S. A. N. Nouwens, B. de Jager, M. M. Paulides, W. P. M. H. Heemels

Model predictive control (MPC) is an optimization-based control strategy with broad industrial adoption. Unfortunately, the required computation time to solve the receding-horizon MPC optimization problem can become prohibitively large for many applications with a large number of state constraints. This large number of state constraints can, for instance, originate from spatially discretizing a partial differential equation of which the solution has to satisfy constraints over the full spatial domain. This is particularly the case in MPC for RF-based hyperthermia cancer treatments, which forms a strong motivation for this study. To address this problem, we propose a novel constraint-adaptive MPC framework for linear discrete-time systems. In this framework, we select at each time-step a subset of the state constraints that are included in the optimization problem, thereby reducing the online computational burden. Critically, our framework guarantees the same closed-loop performance, recursive feasibility, and constraint satisfaction properties as the original (non-reduced) MPC scheme. We achieve this result by efficiently exploiting reachable set computations and the MPC cost function. We will demonstrate our novel method using a hyperthermia cancer treatment case study showing a two-orders of magnitude improvement in computation time, with identical closed-loop performance as the original (non-reduced) MPC scheme.

comment: 6 pages, 3 figures, CDC 2021 conference

★ Exploiting Data Centres and Local Energy Communities Synergies for Market Participation

Ángel Paredes, Yihong Zhou, Chaimaa Essayeh, José A. Aguado, Thomas Morstyn

The evolving energy landscape has propelled energy communities to the forefront of modern energy management. However, existing research has yet to explore the potential synergies between data centres and energy communities, necessitating an assessment on their collective capabilities for cost efficiency, waste heat optimisation, and market participation. This paper presents a mixed integer linear programming model to assess the collaborative performance of energy communities, data centres and energy markets. The evaluation focuses on the efficient use of waste heat and the flexibility of job scheduling while minimising system energy costs and maintaining quality of service requirements for data centres. Our results, based on realistic profiles of an energy community and a data centre, showcase significant benefits of these synergies, with a 38% reduction in operating costs and an 87% decrease in heat demand.

comment: Submitted to IEEE PES ISGT Europe 2024

★ Risk-sensitive Affine Control Synthesis for Stationary LTI Systems

Yang Hu, Shahriar Talebi, Na Li

To address deviations from expected performance in stochastic systems, we propose a risk-sensitive control synthesis method to minimize certain risk measures over the limiting stationary distribution. Specifically, we extend Worst-case Conditional Value-at-Risk (W-CVaR) optimization for Linear Time-invariant (LTI) systems to handle nonzero-mean noise and affine controllers, using only the first and second moments of noise, which enhances robustness against model uncertainty. Highlighting the strong coupling between the linear and bias terms of the controller, we reformulate the synthesis problem as a Bilinear Matrix Inequality (BMI), and propose an alternating optimization algorithm with guaranteed convergence. Finally, we demonstrate the numerical performance of our approach in two representative settings, which shows that the proposed algorithm successfully synthesizes risk-sensitive controllers that outperform the na\"ive LQR baseline.

comment: 8 pages, 4 figures, 2 illustrations

★ Real-time Vehicle-to-Vehicle Communication Based Network Cooperative Control System through Distributed Database and Multimodal Perception: Demonstrated in Crossroads

Xinwen Zhu, Zihao Li, Yuxuan Jiang, Jiazhen Xu, Jie Wang, Xuyang Bai

The autonomous driving industry is rapidly advancing, with Vehicle-to-Vehicle (V2V) communication systems highlighting as a key component of enhanced road safety and traffic efficiency. This paper introduces a novel Real-time Vehicle-to-Vehicle Communication Based Network Cooperative Control System (VVCCS), designed to revolutionize macro-scope traffic planning and collision avoidance in autonomous driving. Implemented on Quanser Car (Qcar) hardware platform, our system integrates the distributed databases into individual autonomous vehicles and an optional central server. We also developed a comprehensive multi-modal perception system with multi-objective tracking and radar sensing. Through a demonstration within a physical crossroad environment, our system showcases its potential to be applied in congested and complex urban environments.

comment: ICICT 2024, 18 pages

★ LEADS: Lightweight Embedded Assisted Driving System

Tianhao Fu, Querobin Mascarenhas, Andrew Forti

With the rapid development of electric vehicles, formula races that face high school and university students have become more popular than ever as the threshold for design and manufacturing has been lowered. In many cases, we see teams inspired by or directly using toolkits and technologies inherited from standardized commercial vehicles. These architectures are usually overly complicated for amateur applications like the races. In order to improve the efficiency and simplify the development of instrumentation, control, and analysis systems, we propose LEADS (Lightweight Embedded Assisted Driving System), a dedicated solution for such scenarios.

comment: 12 pages

★ Electric Grid Topology and Admittance Estimation: Quantifying Phasor-based Measurement Requirements

Norak Rin, Iman Shames, Ian R. Petersen, Elizabeth L. Ratnam

In this paper, we quantify voltage and current phasor-based measurement requirements for the unique identification of the electric grid topology and admittance parameters. Our approach is underpinned by the concept of a rigidity matrix that has been extensively studied in graph rigidity theory. Specifically, we show that the rank of the rigidity matrix is the same as that of a voltage coefficient matrix in a corresponding electric power system. Accordingly, we show that there is a minimum number of measurements required to uniquely identify the admittance matrix and corresponding grid topology. By means of a numerical example on the IEEE 4-node radial network, we demonstrate that our approach is suitable for applications in electric power grids.

♻ ★ SPARC: Prediction-Based Safe Control for Coupled Controllable and Uncontrollable Agents with Conformal Predictions

Shuqi Wang, Shaoyuan Li, Xiang Yin

We investigate the problem of safe control synthesis for systems operating in environments with uncontrollable agents whose dynamics are unknown but coupled with those of the controlled system. This scenario naturally arises in various applications, such as autonomous driving and human-robot collaboration, where the behavior of uncontrollable agents, like pedestrians, cannot be directly controlled but is influenced by the actions of the autonomous vehicle or robot. In this paper, we present SPARC (Safe Prediction-Based Robust Controller for Coupled Agents), a novel framework designed to ensure safe control in the presence of coupled uncontrollable agents. SPARC leverages conformal prediction to quantify uncertainty in data-driven prediction of agent behavior. Particularly, we introduce a joint distribution-based approach to account for the coupled dynamics of the controlled system and uncontrollable agents. By integrating the control barrier function (CBF) technique, SPARC provides provable safety guarantees at a high confidence level. We illustrate our framework with a case study involving an autonomous driving scenario with walking pedestrians.

comment: It's not complete yet

♻ ★ Addressing Trust Issues for Vehicle to Grid in Distributed Power Grids Using Blockchains

Yunwang Chen, Xiang Lei, Linni Jian

While blockchain offers inherent security, trust issues among stakeholders in vehicle-to-grid (V2G) applications remain unresolved due to a lack of regulatory frameworks and standardization. Additionally, a tailored decentralized privacy-preserved coordination scheme for blockchain in V2G networks is needed to ensure user privacy and efficient energy transactions. This paper proposes a V2G trading and coordination scheme tailored to the decentralized nature of blockchain as well as the interests of stakeholders utilizing smart charging points (SCPs) and Stackelberg game model. Case studies using real-world data from Southern University of Science and Technology demonstrate the efficacy of proposed scheme in reducing EV charging costs and the potential for supporting auxiliary grid services.

comment: This paper has been accepted by The 14th International Conference on Power and Energy Systems (ICPES 2024)

♻ ★ Certifiably Robust Policies for Uncertain Parametric Environments

Yannik Schnitzer, Alessandro Abate, David Parker

We present a data-driven approach for producing policies that are provably robust across unknown stochastic environments. Existing approaches can learn models of a single environment as an interval Markov decision processes (IMDP) and produce a robust policy with a probably approximately correct (PAC) guarantee on its performance. However these are unable to reason about the impact of environmental parameters underlying the uncertainty. We propose a framework based on parametric Markov decision processes (MDPs) with unknown distributions over parameters. We learn and analyse IMDPs for a set of unknown sample environments induced by parameters. The key challenge is then to produce meaningful performance guarantees that combine the two layers of uncertainty: (1) multiple environments induced by parameters with an unknown distribution; (2) unknown induced environments which are approximated by IMDPs. We present a novel approach based on scenario optimisation that yields a single PAC guarantee quantifying the risk level for which a specified performance level can be assured in unseen environments, plus a means to trade-off risk and performance. We implement and evaluate our framework using multiple robust policy generation methods on a range of benchmarks. We show that our approach produces tight bounds on a policy's performance with high confidence.

♻ ★ Posterior Sampling-based Online Learning for Episodic POMDPs

Dengwang Tang, Dongze Ye, Rahul Jain, Ashutosh Nayyar, Pierluigi Nuzzo

Learning in POMDPs is known to be significantly harder than in MDPs. In this paper, we consider the online learning problem for episodic POMDPs with unknown transition and observation models. We propose a Posterior Sampling-based reinforcement learning algorithm for POMDPs (PS4POMDPs), which is much simpler and more implementable compared to state-of-the-art optimism-based online learning algorithms for POMDPs. We show that the Bayesian regret of the proposed algorithm scales as the square root of the number of episodes and is polynomial in the other parameters. In a general setting, the regret scales exponentially in the horizon length $H$, and we show that this is inevitable by providing a lower bound. However, when the POMDP is undercomplete and weakly revealing (a common assumption in the recent literature), we establish a polynomial Bayesian regret bound. We finally propose a posterior sampling algorithm for multi-agent POMDPs, and show it too has sublinear regret.

comment: 41 pages, 9 figures

♻ ★ Mitigating Information Asymmetry in Two-Stage Contracts with Non-Myopic Agents

Munther A. Dahleh, Thibaut Horel, M. Umar B. Niazi

We consider a Stackelberg game in which a principal (she) establishes a two-stage contract with a non-myopic agent (he) whose type is unknown. The contract takes the form of an incentive function mapping the agent's first-stage action to his second-stage incentive. While the first-stage action reveals the agent's type under truthful play, a non-myopic agent could benefit from portraying a false type in the first stage to obtain a larger incentive in the second stage. The challenge is thus for the principal to design the incentive function so as to induce truthful play. We show that this is only possible with a constant, non-reactive incentive functions when the type space is continuous, whereas it can be achieved with reactive functions for discrete types. Additionally, we show that introducing an adjustment mechanism that penalizes inconsistent behavior across both stages allows the principal to design more flexible incentive functions.

comment: To appear in the Proceedings of the 5th IFAC Workshop on Cyber-Physical Human Systems